Design strategies for GitOps repositories

Left portion of picture is a network topology of a Kubernetes cluster with 3 notes, with each node backed by a virtual machine. The right-side shows a tentative mapping of that topology to a folder structure.
Mapping a semantic network to a folder structure is not always straightforward.

Lesson #1: Beware the topology

Do not spend too much time developing an elaborate folder structure resembling the structure of the deployment environment. Anything over a certain depth makes navigation cumbersome, documentation more complicated, and explanations longer.

Figure of system administrator overwhelmed with a folder structure that is over 6 levels deep, containing a deployment environment, IaaS region, infrastructure type, k8s clusters, services, and others.
“Tall” repository trees can make repository navigation a constant chore.

Lesson #2: Versioning with branches and tags

All changes must eventually converge to the main branch, but iterations should be versioned using branches and tags in the repository.

Developer submitting version 1.1.0 to the GitOps repository on a pipeline with “dev”, “test”, “stage”, and “prod” stages. The testing is currently happening in the “test” stage and the remaining stages are still at version 1.0.9.
A new candidate patch branch is being rolled across the deployment pipeline.

Lesson #3: Isolate environment-specific settings

Not all differences are created equal. A core design principle for a continuous deployment pipeline is minimizing differences between stages because fewer differences increase the chances that a change works in the next pipeline stage.

  • Versioning differences. These are the expected transient differences while changes are validated in the previous pipeline stage, such as image URLs for the containers used in the product or versions of micro-services comprising the whole product.
Box on the left represents a pull request with changes in the number of replicas for a service in the “stage” environment and a similar change to the same service in the production zone named “us-south”. A pipeline on the right shows how the change produces no changes to the “dev” and “test” environments, while a robot works on making the changes to the “stage” environment”, and then to the “us-south” environment.
A pull request may contain only changes that are intrinsic to certain environments, producing no results in some portions of the pipeline. As a rule, avoid mixing changes to intrinsic and versioning differences in the same pull request.

Modifying all deployments

  1. The genesis. Developer clones the repository and checkout the “main” branch. Usually, this should be the default branch for the repository.
  2. The developer creates a new branch named “feature-1”, makes local changes to the cloned repository, commits the changes, and pushes the new “feature-1” branch to the remote Git repository at the Git Provider.
  3. The developer submits a pull request for “feature-1” into the “main” branch (For GitLab users, that would be a merge request.)
  4. Trying times ahead. The Git provider issues an event about the pull request to an automated process — owned by the repository owners — responsible for applying the change to the “dev” environment.
  5. The automated process applies the changes in “feature-1” to the development environment and starts the validation tests.
  6. A new branch is born. Assuming the validation tests pass, the new “feature-1” is considered good enough to be merged back into “main,” and that new version of “main” with the changes becomes the basis of a new versioned branched. Note: An automated process cannot guess the correct name for the new branch in a semantic versioning scheme, so it is helpful to allow the author to add that type of clue in the text of the pull request.
  7. Failure is an option. It is still possible that validation in the next pipeline stages uncovers a problem with the new branch resulting in it being sidelined as failed, stopping its progression across the pipeline.
  8. Retreating is not an option. While there are proponents of a rollback strategy for the stages containing a failed branch, accept that rolling forward is the better approach (A topic I covered in a separate article about CI/CD pipelines for GitOps.) In the “roll forward” strategy, a new issue is automatically created, and the team works on a new patch that addresses the issue. Fixing this situation takes precedence over any other modification, and the pipeline is closed for further changes.
  9. If it doesn’t fail, keep it going. With the pull request merged into “main,” that request is closed automatically, and an automated process starts to deploy the changes to the next stage of the pipeline.

Modifying a single environment

If environment-specific folders and files are isolated to small corners of the repository, this scenario is virtually identical to the modification of all environments.

Lesson #4: Beware the Monorepo

“Monorepo” is a strategy where the entire infrastructure is represented in a single Git repository, such as applications, services, and infrastructure.

Monorepo GitOps repo with multiple disciplines, from applications to infrastructure

Lesson #5: The workflow rule. Workflows rule.

This lesson extends the previous one: if monorepos have limited application, it follows that one should always have multiple repos, but which ones? While we examine this lesson, it is also important to note that an excessive proliferation of repositories also introduces new costs in managing all these repositories, keeping tabs on access control lists, webhooks, access tokens, and many others.

People and automation looking at Git repository and collaborating to update a deployment environment.
People and processes shape the Git repository structure, not the deployment topology.

Conclusion

Working with deployment environments and Git repositories is a constant exercise in mapping complex topologies to simpler structures. While a deployment structure may seem like a tempting starting point due to the team’s familiarity, topologies tend to be a misleading organizing principle.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Denilson N.

Denilson N.

Operations architect, corporate observer, software engineer, inventor. @dnastacio