GitOps: git as the source of truth

Why

In ordinary CI/CD, the pipeline itself connects to the cluster and applies changes: kubectl apply, helm upgrade. That's a push model: state gets pushed in from outside. It has two weak spots. The pipeline needs access to the cluster, meaning broad permissions from outside. And nobody watches whether the cluster still has exactly what was deployed: a manual change via kubectl edit lives on until someone notices it.

GitOps flips this into a pull model. An agent (Argo CD, Flux) runs next to the cluster. It watches a git repository with manifests itself and pulls in changes. The pipeline no longer needs access to the cluster.

Concepts

GitOps is a delivery method where the desired state of a system is described in git, and an agent brings the system to match it on its own. The OpenGitOps project states this as four principles:

• state is described declaratively (manifests, not a script of commands);
• it's stored with versions and full history, meaning in git;
• the agent pulls the description from the source itself (pull);
• the agent continuously compares the actual state with the desired one and fixes the difference.

Desired state is what sits in git on the right branch. Actual state is what's really running in the cluster.

Sync is the agent applying manifests from git to the cluster. After it, the application's status in Argo CD is Synced. If the cluster and git diverge, the status is OutOfSync.

Drift is the cluster diverging from git because of a change that bypassed git: someone ran kubectl edit, kubectl scale, or changed a resource some other way.

How it works

Git as the source of truth

The main principle: the cluster's desired state is described entirely in git. To roll out a new version, you make a commit (or merge into the right branch) that changes the manifest. The agent sees the change and applies it.

This has a few consequences:

• A rollout is a git commit, with an author, a review, and a history.
• A rollback happens via git revert: a new commit restores the old manifest, and the agent restores the cluster. History isn't rewritten in the process, so git push --force doesn't work for a rollback.
• No outside access to the cluster is needed: the permissions belong to the agent inside.

The agent doesn't learn about a commit instantly. By default, Argo CD polls the git repository roughly once every 3 minutes (120 seconds plus a random addition of up to 60 seconds), and for a fast reaction people set up a webhook from GitLab or GitHub. In Flux the interval is set on each Kustomization object through the spec.interval field.

Drift and how to fix it

The agent doesn't just apply new commits. It regularly compares the desired state (git) with the actual one (the cluster). If someone changed a resource bypassing git, the agent sees the drift and shows the difference.

What the agent does about drift depends on the setting:

• Without auto-sync, Argo CD only marks the application OutOfSync. To apply a new commit or clear the drift, press Sync or run argocd app sync.
• With auto-sync (syncPolicy.automated), Argo CD applies new commits on its own. By default it doesn't touch manual changes in the cluster though: the status is OutOfSync, and the change lives on until the next commit to that application.
• With selfHeal: true, Argo CD restores the cluster to match git even for a manual change. Then changes made outside git don't survive.

Flux behaves like selfHeal mode: on every interval the kustomize-controller compares the cluster with git and fixes any drift it finds.

What do you do if a manual change was actually needed (say, an on-call engineer raised a limit overnight so a service wouldn't crash)? Move it into git with an ordinary commit. Then the cluster and git match again, and the change gets an author and a history.

What GitOps doesn't do

GitOps is responsible for delivery (continuous delivery), not for building. The image is still built by an ordinary CI: tests, docker build, a push to the registry. GitOps kicks in next, when a new image needs to be rolled out to the cluster.

A typical setup: CI builds the image and updates the tag in the git repository with manifests, and the GitOps agent picks that up and rolls it out to the cluster.

Common mistakes

• Fixing the cluster by hand and forgetting about git. With selfHeal, the next reconcile silently wipes the change; without selfHeal, the next commit wipes it. Always carry a needed change through to git.
• Rolling back with git reset and git push --force. The agent will roll back the cluster, but the release itself disappears from history, along with the trace of what went wrong. Roll back with git revert.
• Expecting that deleting a file from git deletes the resource. By default, auto-sync in Argo CD doesn't delete resources; prune: true turns that on. In Flux, deletion is controlled by the spec.prune field of the Kustomization object.
• Putting secrets into git as plain text. Everyone with access to the manifest repository can read it. For secrets, people use Sealed Secrets, SOPS, or External Secrets Operator.
• Mixing up GitOps with CI. The agent doesn't build images or run tests. If the image tag in git wasn't updated, the agent won't roll out anything new.

What you'll need for the tasks

• git -C app add, git -C app commit -m "...", git -C app push origin main
• git -C app revert --no-edit HEAD and git -C app log --oneline
• docker logs <container>: the agent's log, with Synced and OutOfSync lines
• curl -s http://127.0.0.1:PORT/file, to see what's "deployed"

In the labs, a training agent stands in for Argo CD, and a web server stands in for the cluster. The cycle is the same: the agent pulls the main branch, compares it with the actual state, and applies the difference.

In the tracks ahead this connects to Kubernetes: the manifests from git that the agent rolls out are the same Deployment and Service objects.