Designing a GitOps Promotion Pipeline for Multi-Cluster Kubernetes with ArgoCD

TL;DR: ArgoCD's job is reconciliation — making a cluster match what Git says it should be — not deciding when a change is ready to move from dev to staging to production. Conflating those two responsibilities is the single most common mistake in multi-cluster ArgoCD setups: teams let production auto-sync directly from main and lose the approval gate entirely, or they build a promotion mechanism by hand that duplicates half of what a purpose-built promotion tool (Kargo, or a CI-driven PR-based flow) already does well. This piece walks through the architecture that separates the two concerns cleanly: ApplicationSets for declaring what runs where, and an explicit, auditable promotion path for deciding when a version is allowed to move.

Two different jobs that get bundled into one tool

ArgoCD continuously reconciles: it watches a Git path, compares it to live cluster state, and either auto-syncs or flags drift. That's a reconciliation loop, and it's excellent at it — you get drift detection, rollback via git revert, and a full audit trail of every change as a commit. What it deliberately does not do on its own is decide when a given commit or image tag should be considered production-ready. That's a promotion decision, and it needs its own gate: automated checks, a human approval, a canary analysis window, or all three.

The failure pattern to avoid: a single Application pointing production's overlay directly at main, with auto-sync enabled. Every merge to main is now a de facto production deploy, and the "promotion pipeline" is just "hope the PR review was thorough enough." This works until it doesn't — usually at 2am, from a change that looked trivial in the diff.

The building block: separating "where" from "when"

App-of-Apps: hierarchy without automatic scale

The App-of-Apps pattern — one parent Application whose sync target is a directory of child Application manifests — gives you a single entry point that fans out to everything else. It's a clean mental model for organizing a handful of services. It stops being clean past roughly a dozen child apps: sync waves get hard to reason about, a change to the parent can trigger cascading syncs across unrelated children, and there's no per-cluster templating built in — you're hand-maintaining near-duplicate manifests per environment.

ApplicationSets: the right tool for "many clusters, one template"

ApplicationSets solve the templating problem the App-of-Apps pattern doesn't. A generator produces a set of parameters (one entry per cluster, per environment, per Git directory — whatever axis you're fanning out on), and a single Application template is rendered once per generated entry.

The Cluster generator is the one that matters for multi-cluster fleets: it automatically creates an Application for every cluster registered with ArgoCD, or a filtered subset matching cluster labels.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    metadata:
      name: '{{name}}-platform-addons'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/platform-gitops.git
        targetRevision: main
        path: 'addons/base'
      destination:
        server: '{{server}}'
        namespace: platform-system
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

New cluster onboarding becomes: register the cluster with a production label, and every ApplicationSet using that selector picks it up automatically — no manifest duplication, no forgotten environment. This is the pattern behind "PR creates a cluster CR, cluster gets provisioned, ArgoCD's cluster generator picks it up, addons land automatically" fleet-management workflows that platform teams run at scale in 2026.

For anything with cross-cutting logic — different config per combination of cluster and app, say — the Matrix generator combines two generators (e.g., Cluster × Git directory) and templates on the cartesian product. Start with List or Cluster generators until the template syntax and parameter substitution are second nature; reach for Matrix only once you actually have a two-axis fan-out problem, not preemptively.

Building the actual promotion path

Four implementation options exist for moving a change through dev → staging → production in a GitOps flow. They differ in where the "current version per environment" state lives.

1. Separate directories per environment, same branch

apps/orders-service/
  base/
  overlays/
    dev/
    staging/
    production/

Promotion is a PR that bumps the image tag (or Kustomize patch) in overlays/staging/kustomization.yaml, then a second PR that does the same for overlays/production. Simple, auditable via git log, and the most common starting point. The weakness: nothing stops someone from editing overlays/production directly without having gone through staging first — the ordering is a process convention, not an enforced one, unless you add branch protection rules or a CI check that diffs environments.

2. Separate branches per environment

main → staging → production, with promotion as a merge (often a fast-forward or a cherry-pick) between branches. ArgoCD Applications point at different branches per environment. This makes "what's different between staging and production" a git diff staging production away, which is genuinely useful for audits — but branch-per-environment workflows are notorious for merge conflicts and drift when hotfixes land directly on a downstream branch and never get merged back upstream.

3. Separate repos per environment

Rare, and mostly seen in regulated environments where production Git access needs to be a smaller, more tightly controlled group than the group that can merge to the app repo. A CI job mirrors approved changes from the app repo into the environment-specific repo on promotion. Heaviest to operate; justified only when the access-control requirement is real, not just "feels more secure."

4. Image tag / digest promotion with a dedicated promotion controller

This is where Kargo (built specifically to sit alongside ArgoCD) earns its place: Argo CD keeps making the cluster match Git, while Kargo owns moving an artifact through stages with verification and approval, updating the Git state that ArgoCD then reconciles. Kargo stages model the pipeline explicitly — an image built and tagged, verified in dev via automated checks, promoted to staging pending a health check, promoted to production pending human approval — and each promotion is still, underneath, a Git commit that ArgoCD picks up and syncs. You get explicit gates without hand-rolling a promotion state machine in a CI YAML file.

If you don't want another controller in the cluster, the equivalent can be built with a CI pipeline (GitHub Actions, GitLab CI) that opens the promotion PR automatically after dev verification passes, and requires a manual approval gate (a protected environment, a required reviewer) before merging the PR that updates the production overlay. Functionally similar outcome, more YAML to maintain yourself.

Progressive sync: gating promotion on health, not just approval

ApplicationSets support a strategy: type: RollingSync configuration that groups clusters into ordered steps and only proceeds to the next step once the current step's Applications report healthy:

spec:
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: environment
              operator: In
              values: [canary]
          maxUpdate: 100%
        - matchExpressions:
            - key: environment
              operator: In
              values: [production]
          maxUpdate: 25%
        - matchExpressions:
            - key: environment
              operator: In
              values: [production]
          maxUpdate: 100%

This is the mechanism that turns "deploy to every production cluster simultaneously" into "deploy to the canary cluster, wait for ArgoCD's health check to pass, deploy to 25% of production clusters, wait again, then finish the rollout." Health checks here are Kubernetes-native (pod readiness, custom health checks defined via Lua scripts for CRDs) — they don't inherently know about business metrics like error rate or latency. For metric-gated canary analysis (only proceed if p99 latency and error rate stay within bounds), you need Argo Rollouts in front of the Deployment, with an AnalysisTemplate querying Prometheus, and ArgoCD's health assessment reading the Rollout's status rather than a bare Deployment's.

Failure modes worth designing against up front

Auto-sync with no promotion gate on production. Covered above, but worth repeating because it's the default a team falls into when they set up ArgoCD quickly and never revisit it: automated: {} with no branch or approval boundary between "merged to main" and "running in prod."

Sync waves masking dependency ordering bugs. sync-wave annotations control ordering within a single Application's sync, but they're a blunt tool — they don't express "wait until this Deployment is actually serving traffic," only "apply this manifest before that one." A CRD that needs to be established before a resource using it, or a migration Job that must complete before the app pods start, both need PreSync hooks or Sync hooks with hook-weight, not just wave numbers.

Cluster generator drift when cluster labels are wrong. The entire fleet-management convenience of the Cluster generator depends on labels being correct and current. A cluster mislabeled staging when it's actually serving production traffic silently pulls in the wrong ApplicationSet templates — this is a case for admission-time validation on cluster registration, not just trust.

Secrets in the promoted manifest path. GitOps promotion moves a Git commit; it should never move a plaintext secret. Sealed Secrets, External Secrets Operator pulling from a vault at sync time, or SOPS-encrypted values are the standard answers — the promotion pipeline should only ever move references, never values.

A decision framework

Getting the reconciliation/promotion split right early is far cheaper than retrofitting approval gates onto a fleet that's already auto-syncing to production. Syslabs' cloud infrastructure and platform engineering work regularly starts exactly here — alongside broader DevOps and platform strategy engagements.

Sources:

• Argo CD documentation, "Cluster Generator - ApplicationSets"
• oneuptime.com, "How to Implement Promotion Workflows with ArgoCD" and "How to Use Progressive Syncs in ArgoCD ApplicationSets"
• akuity.io, "GitOps Best Practices: A Complete Guide (2026 Edition)"
• GitHub argoproj/argo-cd Discussion #5667, "Best practices for promotion between clusters"