GitOps repos: Five lessons on what to include and what to skip

Lesson #1: Configuration formats matter

Part of the decision about which configuration to manage with GitOps is the choice of the IaC framework, which should prioritize alignment with the system abstractions.

Two-part picture. Left side has a repo administrator staring at a circle representing the entire system. The circle contains a directed graph with circles and squares containing single alphabet letters. Right-side of picture has a tree-like structure of folders, containing two parent folders (VMs and clusters,) each containing a few alphabet letters mapping to the letters in the left-side of the picture. Robots labeled “Terraform” and “Argo” look at the “VMs” and “clusters” folders, respectively
It may be impractical to use a single IaC framework in a large environment due to the specialized needs of complex components. For instance, Terraform excels at allocating resources at the IaaS level, while Argo CD excels at managing Kubernetes configuration.

Lesson #2: Favor behavior before state

Modern infrastructure can be mind-bogglingly extensive, making the complete representation of their desired state virtually impossible. We often need to settle for things that act the same instead of obsessing over making them precisely the same.

Folder structure with “VMs” and “clusters” at the first level. The “VMs” folder has a highlighted “G” folder with a callout showing its definition being “Ubunty 16.04” plus changes to files in “/etc/hosts” plus an invocation of Terraform’s “remote-exec” primitive.
GitOps principles call for declarative and immutable settings. Still, sometimes you need to bend those principles and store configuration relative to baselines that may be entirely or partially outside your control.

Lesson #3: Leave out state co-managed by specialized components

The first type of “state” matching this lesson is application data. It is managed inside specialized servers with their own APIs and tooling. Now, let’s broaden the concept from “application data” to “data that has its dedicated system of record, lifecycle, and workflow.”

A circle represents the whole environment, containing a directed graph. This is the same circle depicted atop the article, but now the smaller circle labeled “F” is at the edge of the larger circle, and a robot inspects the configuration for the smaller circle and then sets the number of “replicas” of the circle to “5”. An admin responsible for the whole environment watches the robot and allows it  to proceed.
Automated processes and components may be better positioned to configure or tune details of some aspects of the infrastructure.

Lesson #4: Avoid secrets and certificates

We often see articles and tutorials extolling the virtues of storing encrypted secrets inside Git repositories, a technique commonly known as “sealed secrets.” Since secrets (and certificates) have their own system of record, lifecycle, and workflows, the previous lesson of avoiding that kind of data in a Git repository still applies.

Lesson #5: Treat the GitOps CI/CD pipeline as code too

I earlier wrote about how it is possible to bring the desired state of the entire system into Git repositories. This section extends the definition of “system” to the GitOps repositories and pipelines underpinning those deployments.

The entire overview picture at the beginning of the article is depicted inside a larger circle, representing the whole system. On the right side of the entire picture has a robot tending to a tree structure containing top-level folders for pipeline-as-code technologies, such as Argo and Tekton. These folders contain the pipeline definitions for the GitOps folders.
GitOps can also manage the artifacts behind the primary GitOps process, supporting versioned and consistent settings for CI/CD pipelines.


Define the boundaries of your entire system early on and decide on a phased approach to bring configuration into one or more GitOps repositories. Some components will be whole systems within the system, requiring multiple GitOps frameworks to cover everything.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Denilson N.

Denilson N.

Operations architect, corporate observer, software engineer, inventor. @dnastacio