Share this letter & I’ll send you some rewards for the referrals.
The CI/CD pipeline changed how we deploy software.
It made deployments faster, repeatable, and automated. But once a deployment finishes, the pipeline has no idea what happens inside the cluster afterward.
Someone could edit a Deployment with kubectl, apply a hotfix during an incident, or update a resource manually. Those changes bypass Git and the pipeline.
As manual changes increase, the cluster slowly drifts from what gets stored in Git.
At some point, production no longer matches the configuration everyone believes is running. When an issue appears, it becomes difficult to see what changed... Also it becomes difficult to know who changed it or whether the running state still matches the last reviewed configuration.
GitOps solves this by making Git the source of truth for the cluster…not just another step in the deployment process…
An agent continuously compares the live cluster with the desired state stored in Git and reconciles any differences. If someone changes a resource outside Git, ArgoCD detects the drift and brings the cluster back to the expected state.
The CI pipeline still builds, tests, and publishes the application1.
But instead of deploying directly to Kubernetes, it updates Git. ArgoCD then handles the deployment and keeps the cluster aligned with the repository.
§
[Webinar] How to stop rewriting your agent’s code
Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.
But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.
Join us live on Oct 28 (FREE) to see:
The 8-levels of AI maturity and where teams get stuck
How a context engine curates context and resolves conflicts before your agent writes a line of code
Live demo: token savings in the same coding task, and use cases beyond code gen
Bonus: 2 open source tools you can use today.
(Thanks to Unblocked for partnering on this newsletter.)
§
I want to introduce Ayaan as a guest author.
He’s a Certified Kubernetes Administrator (CKA) who writes about Kubernetes, cloud infrastructure, and cloud-native systems. He builds in public and shares practical tips, projects, and lessons learned along the way.
Check out his work & socials:
§
Here’s what’s inside:
Your deployment succeeds. Then production starts drifting. How manual Kubernetes changes create configuration drift, why traditional CI/CD pipelines miss them, and how GitOps keeps your cluster aligned with Git.
Your CI pipeline no longer needs the keys to Kubernetes. How ArgoCD’s pull-based model changes deployments, where credentials live, and what happens when a system gets compromised.
ArgoCD runs a reconciliation loop behind the scenes. How its core components detect changes, compare desired and live states, and use sync policies, self-healing, and hooks to control deployments.
One template, hundreds of applications. How ApplicationSets manage deployments across clusters, why repository structure matters, and how Git commits promote releases from development to production.
GitOps gets harder when you reach production. How to manage secrets, access control, monitoring, progressive delivery, and disaster recovery, plus the tradeoffs you need to understand before adopting it.
§
What Is GitOps
GitOps2 is an operating model where a Git repository is the single source of truth for everything running in your infrastructure.
Every change to your application or cluster configuration happens through a Git commit. It gets reviewed, approved, and merged through the same process as application code.
i.e., changes are not applied directly to the cluster outside Git.
§
GitOps Principles
The OpenGitOps3 working group at CNCF4 defined four principles to distinguish GitOps from a traditional deployment pipeline.
Let’s dive in!
1. Declarative
In a declarative system, you describe what you want, not how to get there.
Instead of writing a script with steps like “create this deployment, then scale it to three replicas, then expose it on port 80,” you write a YAML file describing the end state. The system figures out how to make it happen.
Kubernetes is declarative by nature.
Helm charts5, Kustomize6 overlays, and raw YAML manifests all follow the same model.
This matters for GitOps because a declarative system supports storage, versioning, and comparison. Unlike a sequence of commands, two YAML files allow you to see exactly what changed.
2. Versioned and Immutable
Every change to the desired state is a Git commit.
And Git commits do not change…
This gives you a complete, tamper-evident history of everything your cluster has ever been asked to run. Who requested the change, who approved it, when it got merged, and exactly what it modified. The audit trail becomes part of the deployment workflow, not a separate system.
Rollback is also a natural consequence of this principle. Reverting to a previous state means reverting a commit.
3. Pulled Automatically
This principle changes who is responsible for applying changes.
Instead of an external CI/CD pipeline pushing updates into Kubernetes, an agent inside the cluster pulls the desired state from Git… It then applies the changes.
The security implications of this shift are significant, and they get their own section shortly.
4. Continuously Reconciled
The previous three principles describe how state gets defined and delivered.
This one describes what happens after:
Continuous reconciliation means repeatedly comparing the live cluster state with the desired state in Git. These comparisons detect changes introduced by a manual edit, an autoscaler, or a partial update. The configured sync policy determines whether ArgoCD reports the difference or corrects it automatically.
To understand why GitOps works differently, we first need to compare it with the deployment model most people already use…
§
CD Without GitOps vs CD With GitOps
With the four principles in place, the difference between traditional CD & GitOps is easier to see.
It is not just a philosophical shift. The two models produce different security properties and failure modes.
They also provide different levels of visibility into what runs in production…
How CD Worked Before GitOps
In a traditional push-based setup7, the CI/CD pipeline carries a release from source code to the Kubernetes cluster.
It runs tests, builds a container image, and pushes the image to a registry. The deployment stage then updates Kubernetes using a command such as kubectl apply or helm upgrade.
For example, releasing version 1.5 means the pipeline updates the Deployment to use the new image. Kubernetes then replaces the existing pods according to the configured rollout strategy.
The pipeline directly initiates the deployment. The deployment stage delivers the new configuration to the cluster.
How CD Works With ArgoCD
With ArgoCD8, the pipeline still tests the application, builds the image, and publishes it to a registry. It then updates the image reference in the configuration repository.
For the same version 1.5 release, the pipeline changes the image tag in a Kubernetes manifest, Helm values file, or Kustomize configuration. This change becomes a Git commit rather than a deployment command executed against the cluster.
ArgoCD detects the updated configuration and compares it with the live resources. With automated sync enabled, it applies the update. With manual sync, it waits for an operator to trigger synchronization. Kubernetes then performs the rollout.
i.e., Git becomes the handoff between the two systems.
CI produces the image and records the intended version. ArgoCD handles applying the configuration to Kubernetes.
This separation also changes where deployment credentials are needed, which brings us to the security boundary between push and pull.
§
Push vs Pull: Security Boundary
The shift from push to pull is not just an architectural preference.
It changes where credentials live, how far a compromised system can reach, and the damage a supply chain attack can cause.
Now let’s compare the two approaches:
Push-Based: Pipeline Holds the Keys
In a push-based model, the CI pipeline needs direct access to the cluster to deploy anything.
i.e., cluster credentials live inside the CI system.
Anyone with sufficient access to the CI system or pipeline configuration could also gain access to those credentials. Every third-party action, plugin, or integration running inside the pipeline runs adjacent to them. And because pipelines are triggered by code changes, a compromised dependency runs in the pipeline. It runs in the same environment as your cluster credentials.
A compromised pipeline in this model is effectively a compromised cluster.
The attacker does not need to find a vulnerability in Kubernetes… They just need to get malicious code into the build process & let the pipeline do the rest.
The blast radius is as wide as the credentials allow.
For most engineers, those credentials have broad permissions. Restricting them precisely is operationally painful. A compromised push-based pipeline often means cluster-admin access is in the wrong hands.
Pull-Based: Cluster Holds the Keys
In a pull-based model, the cluster reaches out to Git.
The CI system never touches the cluster.
ArgoCD runs inside the cluster with scoped RBAC9 (Role-Based Access Control) permissions. It pulls the desired state from Git and applies it. The CI pipeline updates the Git repository when a build succeeds, and its job ends there. No cluster credentials ever leave the cluster environment.
A compromised CI system in this model could modify Git.
But modifying Git only changes the desired state. It does not directly execute anything in the cluster. ArgoCD still has to pull the change and reconcile it with the cluster. Because every change goes through Git first, you get a reviewable, auditable record before it reaches production.
And the blast radius10 shrinks dramatically. A compromised pipeline becomes a much less dangerous event.
§
§
ArgoCD Architecture
Let’s look at how ArgoCD is built:
ArgoCD is not an external service you call into.
It runs as Kubernetes workloads inside the cluster it manages, in the argocd namespace. This is intentional. The agent needs direct access to the Kubernetes API to reconcile state. Running inside the cluster provides the access & keeps cluster credentials inside the cluster.
Here are 5 core components that work together to implement the GitOps reconciliation loop:
1. API Server
API Server is the entry point for everything human-facing in ArgoCD.
It exposes both gRPC11 & REST APIs consumed by the ArgoCD UI, argocd CLI, and any external tools integrating with ArgoCD. Every user request, whether it comes from the UI, CLI, or an external integration, passes through the API Server.
Also it handles all authentication & authorization.
Before any request reaches the rest of the system, the API Server verifies who is making it and whether they have permission.
2. Repo Server
The Repo Server does one thing: render Kubernetes manifests from a Git repository.
It clones the repository, reads the source, and generates the manifests ArgoCD will apply to the cluster. It supports plain YAML, Helm charts, Kustomize overlays, and Jsonnet12. Whatever format your configuration lives in, the Repo Server handles the rendering.
It’s deliberately isolated from the Application Controller.
Rendering manifests requires running Helm, Kustomize & Jsonnet. Those processes are often CPU-intensive and occasionally unstable. Keeping the Repo Server separate isolates render failures from the controller’s reconciliation loop.
3. Application Controller
The Application Controller is the heart of ArgoCD…It runs the reconciliation loop.
On every cycle, it gets the desired manifests from the Repo Server and the live cluster state from the Kubernetes API. Then it computes the diff between the two. Based on the application’s sync policy, it either alerts you to the difference or corrects it automatically.
The Application Controller runs as a StatefulSet.
(Scaling and sharding are covered later.)
4. Redis
Redis serves as a cache layer between Application Controller and Kubernetes API.
Without it, every UI load and every reconciliation cycle would need to query the Kubernetes API directly. At scale, this would generate unnecessary load on the Kubernetes API. Redis stores application state and cluster data so repeated reads hit the cache instead of the API.
It is worth being clear about what Redis is and is not in this context.
It is a cache, not a primary data store. ArgoCD does not depend on Redis for correctness. If Redis goes down, ArgoCD degrades gracefully. Performance suffers, but reconciliation continues.
5. Notifications Controller
Notifications Controller watches ArgoCD events & route alerts to external systems.
When an application goes OutOfSync, a sync fails, or a health check degrades, the Notifications Controller sends an alert. It uses the destination you configured, such as Slack, email, PagerDuty, or a webhook. You define triggers and templates, and the controller handles the delivery.
In most production deployments, it gets installed alongside the rest of ArgoCD. It connects ArgoCD’s internal events with the people and systems responsible for responding to them.
Together, these components form ArgoCD's control plane...
Each component has a specific responsibility:
API Server handles user requests.
Repo Server renders manifests.
Application Controller reconciles applications.
Redis caches data to reduce repeated API calls.
Notifications Controller reports important events.
Together, these components keep the cluster aligned with the desired state in Git.
With the architecture covered, let’s see how these components work together during reconciliation…
§
Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.
When you upgrade, you’ll get:
High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability & performance.
(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)