GitOps and CI/CD

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Applying manifests from a laptop with kubectl apply does not scale past one person. GitOps makes a Git repository the source of truth for what should run, and lets an agent inside the cluster pull that state and reconcile it. CI still builds and tests; it stops being the thing that holds cluster credentials. This page is written against Kubernetes 1.37, Argo CD 3.x and Flux 2.x (check the current releases, as both move quickly — see the section index for the version baseline). It delivers the packaging formats from Kustomize and Helm, and builds on the pipeline in CI/CD with GitHub Actions.

The OpenGitOps principles

The OpenGitOps working group (CNCF) defines four principles:

Declarative — the desired state is expressed declaratively.
Versioned and immutable — it is stored in a system that keeps history and prevents silent rewrites (Git).
Pulled automatically — software agents automatically pull the desired state from the source.
Continuously reconciled — agents observe actual state and attempt to apply the desired state.

The operational consequences: a deployment is a commit, a rollback is a revert, an audit log is git log, and the cluster API does not need to be reachable from CI. That last point is a real security benefit, because a pipeline that holds cluster-admin credentials is a high-value target.

Source: OpenGitOps.

Argo CD

Argo CD runs in the cluster and watches one or more Git repositories. The unit of work is an Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bookshelf-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/bookshelf-deploy.git
    targetRevision: main
    path: k8s/overlays/prod        # a Kustomize overlay; Helm charts work too
  destination:
    server: https://kubernetes.default.svc
    namespace: bookshelf
  syncPolicy:
    automated:
      prune: true                  # delete what was removed from Git
      selfHeal: true               # revert manual changes in the cluster
    syncOptions:
      - CreateNamespace=true

Sync is the act of making the cluster match Git. With automated off, Argo CD reports OutOfSync and waits for a person to press sync; with it on, it applies changes itself. selfHeal is the drift-correction switch (see below), and prune controls whether objects deleted from Git are deleted from the cluster.

Source: Argo CD Docs — Application specification.

Flux

Flux is a set of controllers. A GitRepository fetches the source, and a Kustomization (Flux’s own kind, not the same as the kustomization.yaml file) or a HelmRelease applies it:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: bookshelf
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/example/bookshelf-deploy.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: bookshelf-prod
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: bookshelf
  path: ./k8s/overlays/prod
  prune: true
  targetNamespace: bookshelf

To deliver the chart from Helm instead, point a HelmRelease at an OCI HelmRepository or at the chart inside the GitRepository:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: bookshelf
  namespace: bookshelf
spec:
  interval: 10m
  chart:
    spec:
      chart: ./charts/bookshelf
      sourceRef:
        kind: GitRepository
        name: bookshelf
        namespace: flux-system
  values:
    replicaCount: 3

Source: Flux Docs — GitRepositories and Flux Docs — Kustomizations.

Sync and drift detection

Both tools continuously compare the rendered desired state with the live objects. Drift is any difference that did not come from Git — someone ran kubectl scale by hand, or a mutating webhook (see Admission Control and Policy) added a field. The tool either reports it (Argo CD OutOfSync, flux get kustomizations) or reverts it (selfHeal, Flux’s default reconciliation). Fields legitimately managed by another controller, such as spec.replicas under an HPA (see Autoscaling), must be excluded from comparison, otherwise the two controllers fight.

The CI side: GitHub Actions

CI builds and tests the image, pushes it, and then records the new version in Git. This workflow bumps the tag in the Kustomize overlay by committing to the deployment repository:

name: release
on:
  push:
    tags: ["v*"]
permissions:
  contents: read
  packages: write
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      - id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/example/bookshelf-catalog-api:${{ github.ref_name }}
      - name: Bump image in the prod overlay
        run: |
          git clone https://x-access-token:${{ secrets.DEPLOY_REPO_TOKEN }}@github.com/example/bookshelf-deploy.git
          cd bookshelf-deploy/k8s/overlays/prod
          kustomize edit set image \
            ghcr.io/example/bookshelf-catalog-api=ghcr.io/example/bookshelf-catalog-api@${{ steps.build.outputs.digest }}
          git -c user.name=ci -c user.email=ci@example.com commit -am "catalog-api ${{ github.ref_name }}"
          git push

The alternative, for teams that do not use a GitOps agent, is a final step helm upgrade --install bookshelf ./charts/bookshelf -n bookshelf --set image.tag=${{ github.ref_name }} --rollback-on-failure (Helm 4’s name for --atomic) with a kubeconfig stored as a secret; it works, but it is the push-based model GitOps tries to avoid. Keep every credential in the platform’s secret store (${{ secrets.NAME }}) and never in the repository.

Source: GitHub Docs — GitHub Actions and Docker Docs — Docker Build GitHub Actions.

Tag versus digest pinning

A tag such as 1.0.0 is a mutable pointer: the registry can be made to serve a different image under it. A digest (image@sha256:…) names exact bytes. Tags are readable; digests are reproducible and tamper-evident. The usual compromise is to build with a readable tag and let CI write the digest into the overlay, so the cluster runs exactly what was tested. See Image Best Practices and Security.

Environment promotion

Promotion means moving a known-good version from one environment to the next. With the overlay layout, it is a Git operation: CI updates overlays/dev, automated tests run against dev, then a pull request (or a workflow step) copies the same digest into overlays/staging and finally overlays/prod. The pull request gives a review and an approval gate; the agent does the deploying. Keeping the application code repository separate from the deployment repository avoids build loops and gives operations its own review flow.

Secrets in GitOps

Git must not contain plain secrets, yet the desired state includes them. Three common answers, all of which build on Secrets:

Sealed Secrets — encrypt a Secret with the cluster’s public key into a SealedSecret that is safe to commit; only the in-cluster controller can decrypt it.
SOPS — encrypt the values inside a YAML file with a KMS key or age key; Flux decrypts natively, and Argo CD does so through a plugin.
External Secrets Operator — commit only an ExternalSecret that references a value in a cloud secret manager; the operator creates the Secret in the cluster, so no secret material enters Git at all.

The loop