Canary Deployments

Prometheus Operator simplifies the deployment and configuration of Prometheus, a monitoring system and time series database. Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.

Flagger uses Prometheus to retrieve metrics about SecureAuth and uses this information to make decisions during the canary deployment. If the metrics indicate that the canary version is causing issues, Flagger will halt the rollout and revert to the stable version.

To install prometheus stack in the monitoring namespace, use the following commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
helm install prometheus prometheus-community/kube-prometheus-stack --create-namespace --namespace monitoring --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false            

To learn more, visit the Prometheus Operator Helm Chart repository.

Flagger is a Kubernetes operator that automates the promotion of canary deployments using various service mesh providers, including NGINX. Flagger requires a running Kubernetes cluster and uses Prometheus for monitoring and collecting metrics during the canary deployment process. In this step, install Flagger with the NGINX backend:

helm repo add flagger https://flagger.app
helm repo update flagger
helm upgrade -i flagger flagger/flagger --create-namespace --namespace=flagger --set meshProvider=nginx --set metricsServer=http://prometheus-kube-prometheus-prometheus.monitoring:9090
helm upgrade -i loadtester flagger/loadtester --namespace=flagger            

To learn more, visit the Flagger official documentation.

In this step, you need to install SecureAuth on Kubernetes using the kube-acp-stack Helm Chart.

In this step, you'll define a Flagger Canary custom resource. This resource describes the desired state for SecureAuth deployment and plays a key role in enabling the automated canary deployment process.

This custom resource is broken down into the following sections:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: acp
spec:
  provider: nginx
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: acp
  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: acp
  service:
    port: 8443
    targetPort: http
    portDiscovery: true
  analysis:
    interval: 1m
    threshold: 2
    stepWeights: [5, 10, 15, 20, 25, 35, 45, 60, 75, 90]
    webhooks:
      - name: "check alive"
        type: pre-rollout
        url: http://loadtester/
        timeout: 15s
        metadata:
          type: bash
          cmd: "curl -k https://acp-canary.acp:8443/alive"
    metrics:
      - name: "ACP P95 Latency"
        templateRef:
          name: acp-request-duration
        thresholdRange:
          max: 0.25
        interval: 60s
---
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: acp-request-duration
spec:
  provider:
    type: prometheus
    address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
  query: |
        avg(histogram_quantile(0.95, rate(acp_http_duration_seconds_bucket{job="acp-canary"}[{{ interval }}])) > 0) or on() vector(0)
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: acp-canary
spec:
  endpoints:
    - port: metrics
      interval: 10s
  selector:
    matchLabels:
      app.kubernetes.io/name: acp-canary            

Save the above YAML configuration in a file named acp-canary.yaml and apply it to your cluster:

kubectl apply -f acp-canary.yaml --namespace acp

Applying those resource creates a second SecureAuth ingress, along with additional services for the canary and primary deployments. Check if the Canary resource has been successfully initialized by running:

kubectl get canaries --namespace acp

The status should be initialized

To learn more, visit the NGINX Canary Deployments.

In order to access the ingress on kind using the local domain, you need to map the domain to your localhost in your hosts file.

Open the hosts file in a text editor.

sudo vi /etc/hosts            

Add the following line to the file:

127.0.0.1       acp.local            

Save your changes and exit the text editor.

Now, you should be able to access ingress on the kind cluster via https://acp.local:8443 from your browser.

helm upgrade acp cloudentity/kube-acp-stack --namespace acp --set acp.serviceMonitor.enabled=true --set acp.config.data.logging.level=warn            

It starts a canary version of SecureAuth:

kubectl get pods -n acp -l 'app.kubernetes.io/name in (acp, acp-primary)'

NAME                           READY   STATUS    RESTARTS        AGE
acp-69cdf99895-b6xcb           1/1     Running   0               3m6s
acp-primary-868d5889d6-5xj59   1/1     Running   0               11m            

We want to add latency just to the canary pod in order to simulate issues with the deployed new version.

It's also important to note that adding latency this way, affects all individual packets. If a requiest is made of multiple packets, the latency sums up which can be observed later on.

During the canary deployment, Flagger gradually shifts traffic from the old version to the new version of SecureAuth while monitoring the defined metrics. You can observe the progress of the canary deployment by querying the Flagger logs with kubectl.

kubectl -n acp describe canary/acp
kubectl logs flagger-5d4f4f785-gjdt6 --namespace flagger --follow            

We will start a simple curl command to connect to SecureAuth. As we had set latency to 10ms, responses from canary version are delayed. You can observe over time as more and more requests are hitting the canary version of SecureAuth.

If you used canary manifest from previous steps, canary lasts for 10 minutes and progressivly shifts traffic into the canary version starting from 5% in the first iteration to 90% in the last iteration.

while true; do curl -o /dev/null -k -s -w 'Response time: %{time_total}s\n' https://acp.local:8443/health && sleep 1; done

Response time: 0.006966s
Response time: 0.119855s <- canary version (static delay of 10ms)
Response time: 0.068919s            

Now, lets increase the latency even more causing the canary release to fail:

docker exec my-cluster-control-plane tc qdisc add del veth3ab723f4 root
docker exec my-cluster-control-plane tc qdisc add dev veth3ab723f4 root netem delay 100ms            

The times are above 250ms threshold:

while true; do curl -o /dev/null -k -s -w 'Response time: %{time_total}s\n' https://acp.local:8443/health && sleep 1; done
Response time: 0.607907s
Response time: 0.010327s <- primary version (no delay)
Response time: 0.608345s            

If the average latency is above the threshold, you can see the canary process failing as can be observed in the flagger logs:

{"level":"info","ts":"2023-05-18T14:20:58.628Z","caller":"controller/events.go:33","msg":"New revision detected! Scaling up acp.acp","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:21:58.617Z","caller":"controller/events.go:33","msg":"Starting canary analysis for acp.acp","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:21:58.734Z","caller":"controller/events.go:33","msg":"Pre-rollout check check alive passed","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:21:58.898Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 5","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:22:58.761Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 10","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:23:58.755Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 15","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:24:58.754Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 20","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:25:58.747Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 25","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:26:58.751Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 35","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:27:58.751Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 45","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:28:58.752Z","caller":"controller/events.go:33","msg":"Advance acp.acp canary weight 60","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:31:58.617Z","caller":"controller/events.go:45","msg":"Halt acp.acp advancement ACP P95 Latency 1.34 > 0.25","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:32:58.618Z","caller":"controller/events.go:45","msg":"Halt acp.acp advancement ACP P95 Latency 0.97 > 0.25","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:33:58.616Z","caller":"controller/events.go:45","msg":"Rolling back acp.acp failed checks threshold reached 2","canary":"acp.acp"}
{"level":"info","ts":"2023-05-18T14:33:58.749Z","caller":"controller/events.go:45","msg":"Canary failed! Scaling down acp.acp","canary":"acp.acp"}            

Once you're done with your testing, you may want to clean up the resources you've created. Since all our resources are created within the kind cluster, we just need to delete the kind cluster to clean up.

To delete the kind cluster, run the following command:

kind delete cluster --name=my-cluster            

This command deletes the Kind cluster named my-cluster, and with it all the resources within the cluster. Please replace my-cluster with the name of your cluster if it's different. Be aware that this deletes all the resources within the cluster, including any applications or services you've deployed.

Also, remember to clean up any changes you've made to your /etc/hosts file.

Monitoring the rate of various HTTP error codes (5xx) can help identify issues with the new version.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: acp-error-rate
spec:
  provider:
    type: prometheus
    address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
  query: |
    (100 - sum(rate(acp_http_duration_seconds_count{job="acp-canary", status_code!~"5.."}[{{ interval }}])) / sum(rate(acp_http_duration_seconds_count{job="acp-canary"}[{{ interval }}])) * 100) or on() vector(0)            

This is the time taken to serve a request. It can be helpful in detecting performance regressions in the new version.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: acp-request-duration
spec:
  provider:
    type: prometheus
    address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
  query: |
    avg(histogram_quantile(0.95, rate(acp_http_duration_seconds_bucket{job="acp-canary"}[{{ interval }}])) > 0) or on() vector(0)            

This metric represents the number of messages currently pending in the queue. A sudden increase might indicate a problem with processing the queue's messages.

---
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: acp-pending-messages
spec:
  provider:
    type: prometheus
    address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
  query: |
    sum(acp_redis_error_count{job="acp-canary"}) or on() vector(0)            

This metric represents the time it takes to process a message from the queue. Increased processing time can indicate performance issues with the new version.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: acp-lag-messages
spec:
  provider:
    type: prometheus
    address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
  query: |
    avg(histogram_quantile(0.95, rate(acp_redis_lag_seconds_bucket{job="acp-canary"}[{{ interval }}])) > 0) by (group, stream) or on() vector(0)