Kubernetes Rolling Updates Fly Blind: Why Modern Teams Ship Metric-Gated Canaries With Argo Rollouts



Kubernetes' built-in Deployment has exactly one automated deployment strategy, the rolling update, and it is far less careful than most teams assume. It replaces old pods with new ones a few at a time, but the moment a new pod passes its readiness probe it starts receiving full production traffic, and the only thing that will stop the rollout is a pod crashing outright.

What it cannot do is limit the blast radius of a bad release, gate promotion on real-time metrics, or instantly pivot traffic back to the stable version, and for mission-critical workloads that gap is unacceptable (Opsio, 2026). There is simply no way, with a plain Deployment, to say send 5 per cent of traffic to the new version, measure the error rate, and roll back automatically if it degrades (codingprotocols, 2026).

That sentence is exactly what a canary does, and Argo Rollouts is the open-source Kubernetes controller that adds it. It replaces the Deployment with a Rollout resource, shifts traffic gradually while querying your metrics, and promotes or aborts on its own (Argo Rollouts, 2026). This post builds one end to end: the Rollout and its steps, real traffic weighting, an analysis template, wiring analysis into the rollout, and the commands to drive it.

Replace the Deployment with a Rollout

The Rollout resource mirrors a Deployment spec, the same pod template and selector, and adds a strategy.canary block with a sequence of setWeight and pause steps. The Rollout mirrors the Deployment spec and adds the strategy block, and the controller manages two ReplicaSets at once, stable and canary, adjusting them according to that declarative step sequence (Opsio, 2026).

# A Rollout replaces the Deployment and adds a canary strategy
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-api
spec:
  replicas: 5
  selector:
    matchLabels: { app: payments-api }
  template:
    metadata:
      labels: { app: payments-api }
    spec:
      containers:
        - name: api
          image: ghcr.io/org/payments-api:1.4.3
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

Changing the image triggers a new canary ReplicaSet, and the steps take it from 5 per cent of traffic to 100 in stages, pausing at each so that you, or an analysis run, can decide whether to continue.

The trade-off is more moving parts than a Deployment, and Argo Rollouts is another controller to install and operate. The payoff is control you cannot get from a rolling update, so it earns its place on the services where a bad release is genuinely expensive.

A canary runs under real production load, on real infrastructure, with real data, which is exactly what a staging environment cannot give you (Devtron, 2025). The 5 per cent is not a rehearsal. It is the real thing, contained.

Shift traffic by weight, not by counting pods

To send exactly 5 per cent of requests to the canary, you route by weight at the ingress or service mesh, using a stable Service and a canary Service, rather than relying on the ratio of pod counts. For production, mesh-level weighting via Istio or an ingress like NGINX is strongly preferred, because it is independent of pod count and eliminates the sampling error you get at low replica counts (Opsio, 2026).

# Shift real traffic by weight, not by pod count, using the ingress
strategy:
  canary:
    stableService: payments-api-stable   # non-canary pods
    canaryService: payments-api-canary    # the weighted slice
    trafficRouting:
      nginx:
        stableIngress: payments-api
    steps:
      - setWeight: 5
      - pause: { duration: 10m }

The Rollout points at a stable and a canary Service and tells the ingress to split traffic by the current weight, so 5 per cent means 5 per cent of requests, not roughly one pod in twenty.

The trade-off is a dependency: precise weighting needs a supported ingress controller or a service mesh, and without one your canary granularity is limited to pod-count percentages, which is workable but blunt (Opsio, 2026).

Let metrics make the decision: the AnalysisTemplate

An AnalysisTemplate defines the query that decides whether the canary is healthy, for example the share of non-5xx responses from Prometheus, with a threshold and a failure limit. Argo Rollouts queries a provider like Prometheus for error rates and response times, and if the metric stays within threshold the rollout continues, while increased latency or high failure rates automatically halt or revert it (Akuity, 2025).

# Auto-decide from metrics: is the canary healthy enough to proceed?
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.99   # 99% non-5xx
      failureLimit: 3                        # abort after 3 breaches
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="{{args.service}}",status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total{app="{{args.service}}"}[5m]))

The template measures the success rate every minute, and if it drops below the successCondition three times, the analysis fails. That failure is the trigger the rollout uses to abort.

The trade-off is that automated gates are only as trustworthy as the metrics behind them, so this depends on a stable, low-latency metric store; teams without mature observability cannot lean on analysis and are back to manual promotion (Opsio, 2026).

When the analysis fails, Argo Rollouts sets the canary weight back to zero and marks the Rollout Degraded automatically (Argo Rollouts, 2026). The bad version is pulled from live traffic without a human touching anything, which is the entire point: the rollback happens at 5 per cent, not after it has reached everyone.

Wire the analysis into the rollout

You attach the analysis to a step so it runs at a chosen weight. Analysis can run inline as a step, where an AnalysisRun starts when the step is reached and its success or failure decides whether the rollout advances or aborts entirely (Argo Rollouts, 2026).

# Run the analysis at 25%; a failed run aborts and drops weight to 0
steps:
  - setWeight: 5
  - pause: { duration: 10m }
  - setWeight: 25
  - analysis:
      templates:
        - templateName: success-rate
      args:
        - name: service
          value: payments-api-canary
  - setWeight: 50
  - pause: { duration: 10m }
  - setWeight: 100

Here the rollout ramps to 25 per cent, runs the success-rate analysis against the canary Service, and only proceeds to 50 if it passes; a failure aborts and drops traffic back to zero.

The trade-off is that where you place the analysis is a judgement call. Too early and there is too little traffic for a meaningful signal; too late and a bad release has already reached a large slice, so pick the weight where you get statistically useful data without over-exposing users.

Drive it, and know when to promote or abort by hand

Day to day, you trigger a rollout by updating the image, watch it with the plugin, and keep the ability to intervene when judgement beats the metric. The kubectl argo rollouts plugin lets you watch a rollout, promote past a pause or to full, and abort straight back to stable at any point (codingprotocols, 2026).

# Trigger a rollout by updating the image
kubectl argo rollouts set image payments-api \
  api=ghcr.io/org/payments-api:1.4.3 -n production

# Watch the canary progress through its steps
kubectl argo rollouts get rollout payments-api -n production --watch

# Manually advance past a pause, or abort straight back to stable
kubectl argo rollouts promote payments-api -n production
kubectl argo rollouts abort   payments-api -n production

The set image starts the canary, get --watch shows each step and the current weight, promote advances it, and abort instantly reverts to the stable version.

The trade-off is that manual gates are powerful and easy to overuse. If every rollout waits on a person to click promote, you have rebuilt the slow, human-bottlenecked release you were trying to escape, so reserve manual gates for genuinely high-risk changes and let analysis carry the rest.

The part worth sitting with

So the choice is not really canary versus rolling update, it is whether your deploys are watched or merely hoped over. A plain rolling update ships your new version to everyone and waits to see if the pods fall over, which means the first real signal that something is wrong is often your users telling you. A canary with automated analysis inverts that: the new version meets a sliver of real traffic, the metrics decide in minutes whether it is healthy, and if it is not, it is pulled back to zero before most people ever touch it, with no one paged and no dashboard vigil. That safety is not free, it costs you a controller to run, a service mesh or ingress that can weight traffic, and metrics you actually trust. But on any service where a bad release costs real money or real trust, that is a small price for turning a deployment from a leap of faith into a measured, reversible experiment. Ship to five per cent, let the metrics vote, and promote what earns it.

Author note

I am Mohan Gopi, an Associate DevOps Engineer at Frigga Cloud Labs. I work across AWS, GCP, and Azure, with GitHub Actions as the deployment backbone for everything I ship. The pattern I keep seeing is teams that call their rolling update a canary, when a rolling update has no traffic control and no idea whether the new version is healthy, it just swaps pods and moves on. Moving the risky services onto Argo Rollouts changed how those deploys feel: the new version gets five per cent of traffic, an analysis run watches the error rate for me, and a bad release aborts itself back to stable before I have even opened Slack. The piece people underestimate is the metrics, because an automated gate is only as honest as the query behind it, so I spend more time on the AnalysisTemplate than on the rollout steps. My rule is to let analysis carry the routine releases and keep a manual gate only for the genuinely scary ones. If you want to compare rollout steps and analysis queries, I am on LinkedIn → Mohan Gopi.

Post a Comment

Previous Post Next Post