Six critical blindspots while securing Argo CD

System administrator sits at expansive network control center wearing headphones and listening to music. A hacker crouches in a corner typing commands. A robot carries out the commands on a deployment environment without catching the attention of the system administrator.
GitOps control planes have elevated privileges on entire deployment environments, making them a prime target for supply chain attacks.

Applications and Projects

Before diving into risk analysis, it is helpful to briefly mention Argo CD’s basic deployment unit: Applications.

Diagram with big “Project X” box at the center and arrows leading to the concepts contained in an Argo CD project: source repository, whitelisted resources that can be present in Applications in that project, a group of people who can manage the project, and finally, a block of clusters that act as destinations.
An Argo CD project combines user permissions, lists of allowed resources, source repositories, and destination clusters.

Lesson #1: Use a dedicated project for the control plane

This lesson assumes that you manage the Argo CD control plane through GitOps — how else, right?

  • spec.destination.namespace refers to the namespace where the Argo CD instance is running.
  • spec.destinations[0].server refers to the local cluster: https://kubernetes.default.svc
  • spec.destinations[0].namespace refers to the namespace running the Argo CD instance
A diagram with a “Project control-plane” at the center, with an arrow to a “control-gitops” GitOps repository, a reference to the local cluster, and an arrow to a sticky figure sitting at a desk and typing configuration files, representing an Argo administrator.
A dedicated project for managing the Argo CD instance allows administrators to separate applications that require elevated privileges from other applications meant to manage other resources.
  1. Enabling branch protection on the main development branch of the CI pipeline.
  2. Requiring signed commits.
  3. Requiring code reviews before merging branches.
  4. Restricting who can push to the main development branch.
  5. Adding automated processing to lint pull requests, labeling the pull request as sensitive if it contains changes that affect access control, such as roles and role bindings.

Lesson #2: Argo resources are for Argo admins only

Only GitOps repositories owned by Argo CD administrators should contain resources belonging to the argoproj.io API group.

  1. From the previous point, a Project containing Application resources must include the namespace of the Argo CD instance in its destinations field (see this other discussion.)
  2. Since that destinations field applies to the contents of all repositories in the project, only Argo CD admins or delegates can merge pull requests in those repositories.

Lesson #3: Delete the “default” project

Argo CD’s initial configuration has a “default” project. This project allows authorized users to add applications from any source repository and manage all resources on all destinations.

Diagram with a big box labeled “Project default”, referencing multiple repositories with a bomb icon next to them and a sticky figure with a mask and in a suspicious manner. The project also has an arrow pointing at destinations, with some of them containing the same bomb icon seen in the source repositories.
The “default” application project in Argo CD allows applications from all sources to manage resources on all destinations. It is meant solely for learning purposes in a disposable environment.

Lesson #4: Block ClusterRoleBindings in (most) projects

The previous lessons prevent unauthorized repository owners from commandeering an Argo CD instance. This lesson is about protecting the cluster itself.

Lesson #5: Narrow roles on remote clusters

Argo CD offers distinct deployment topologies where an instance can control anything from the local cluster to a fleet of remote clusters.

  1. Ask the Kubernetes administrator to create a dedicated service account in each cluster, such as “argocd-control-plane-account.”
  2. Ask the Kubernetes administrator to bind that service account to a role or roles) that match the specific namespaces discussed in the previous steps. Full permissions in the entire namespace may be acceptable. Still, one may consider narrower permissions depending on the scenario (for instance, not granting permissions to read secret resources.)
  3. Ask the Kubernetes administrator to generate the “kubeconfig” file from that service account, then use that file when adding the cluster to Argo CD.
  4. Agree on the rotation interval for the service account tokens and the process to replace them in Argo.
Two-part figure. On the left, a Kubernetes administrator stares at 3 boxes, representing clusters. Each box has a hole with a different shape, representing portions of the the cluster that will be managed with GitOps. On the right, the Kubernetes administrator carries a stack of papers representing credentials for those three portions, with “90 days” written on top of the stack. A sitting Argo CD administrator uses those instructions to type commands starting with “argocd cluster add”.
Cluster administrators should create dedicated service accounts with the narrowest possible roles before handing “kubeconfig” files to Argo CD administrators.

Lesson #6: Have a CVE response plan ready

Like any software component, GitOps frameworks may suffer from vulnerability and exposures, and mitigating those risks requires awareness and speed.

Left side of figure has sticky figure with a megaphone announces a new CVE. Right side of figure has administrators frantically trying to apply mitigations and patches.
Once a CVE is announced, administrators should be ready to react quickly with patches and mitigations.

Summary

Argo CD’s out-of-the-box permissions are tuned for a quick on-ramp into learning exercises. They are unsuited for production environments and compounded by many early adopters not realizing the security implications of using the popular “App of Apps” pattern.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Denilson N.

Denilson N.

526 Followers

Operations architect, corporate observer, software engineer, inventor. @dnastacio