Effective CI/CD strategies for updating GitOps repositories

Efficient CI/CD processes requests independently; developers only worry about submitting pull requests.

Lesson #1: Infrastructure is not an app. A declarative way of thinking

The challenge with CI/CD for a GitOps repository is that there is no self-contained binary or package to be transferred over to a runtime environment. The repository has configuration files and instructions for a third-party agent to effect changes in the final medium: the infrastructure.

A pull request is more than its contents: it is a set of instructions for a remote agent responsible for acting on the infrastructure.
  • A separate folder (or configuration file) for each cluster containing only the settings that are exclusive to the cluster.

Lesson #2: Enforce small, independent changes

This lesson affects the continuous integration portion of a CI/CD pipeline. Continuous integration aims at merging frequent code iterations to the main development branch. The foundation of continuous integration is the concept of small, independent code changes validated by comprehensive test automation.

Large pull requests can exceed the unwritten pipeline specifications, resulting in “terraforming”-like reconciliation with the target infrastructure.

Lesson #3: Beware deceptively small changes in pull requests

Changing a few lines on a blueprint may have an outsized impact on the final product in the real world. Likewise, a few characters in a GitOps repository may trigger massive shifts in the infrastructure, so we need to extend the concept of “small, independent changes” to the impact of a change.

Pull requests small in size may contain subtle changes that have an outsized impact on the infrastructure.

Lesson #4: Unit testing is integration testing

One of the cornerstones of continuous delivery is an extensive battery of automated tests. Good test automation covers the range of unit, integration, and system tests. With GitOps, unit testing is more challenging because the resource contents are often proxies to infrastructure components. The resources also lean towards declarative approaches using “mini languages,” such as Terraform files and the occasional DDL file for relational databases.

Applying pull requests to existing environments is often faster than creating entire environments from scratch.

Lesson #5: There is no rolling back

This lesson assumes the GitOps repository is versioned with branches or tags, as outlined in the previous article in this series.

  1. Reverse the pipeline. Roll back the changes on all other pipeline stages to match the rolled back state of the production environment. I consider this approach unthinkable as a standing practice because it is always possible that a completely unrelated pull request has already started its way through the pipeline. Even if one successfully reverses the entire pipeline back to a known baseline, there is still the unresolved matter of dealing with previously merged requests after ejecting them from the pipeline.
When developers and operations managers need to step in to deal with a troubled pull request, the flow of activities comes to a halt, and there is uncertainty on how to progress.
  • One pull request at a time. Assuming the volume of changes allows for it, implement a policy of allowing a single pull request to roll across the pipeline. The pipeline can queue the processing of another pull request until it promotes the current pull request to production.
  • Treat rollbacks as exceptions. Unless the infrastructure supports reliable downgrades to the previous version, do not entertain rollbacks as a regular, automated process. If the infrastructure has complex internal states or, worse, databases, rollbacks rarely bring the system back to health.

Conclusion

CI/CD practices for application development are well-suited for GitOps repositories, but the pipelines produce very different artifacts from applications.

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Denilson N.

Denilson N.

526 Followers

Operations architect, corporate observer, software engineer, inventor. @dnastacio