Six critical blindspots while securing Argo CD

System administrator sits at expansive network control center wearing headphones and listening to music. A hacker crouches in a corner typing commands. A robot carries out the commands on a deployment environment without catching the attention of the system administrator.
GitOps control planes have elevated privileges on entire deployment environments, making them a prime target for supply chain attacks.

Applications and Projects

Diagram with big “Project X” box at the center and arrows leading to the concepts contained in an Argo CD project: source repository, whitelisted resources that can be present in Applications in that project, a group of people who can manage the project, and finally, a block of clusters that act as destinations.
An Argo CD project combines user permissions, lists of allowed resources, source repositories, and destination clusters.

Lesson #1: Use a dedicated project for the control plane

  • spec.destination.server refers to the local cluster: https://kubernetes.default.svc
  • spec.destination.namespace refers to the namespace where the Argo CD instance is running.
  • spec.sourceRepos[] refers to the Git repositories managed by the instance owners and only those repositories containing the control plane’s configuration.
  • spec.destinations[0].server refers to the local cluster: https://kubernetes.default.svc
  • spec.destinations[0].namespace refers to the namespace running the Argo CD instance
A diagram with a “Project control-plane” at the center, with an arrow to a “control-gitops” GitOps repository, a reference to the local cluster, and an arrow to a sticky figure sitting at a desk and typing configuration files, representing an Argo administrator.
A dedicated project for managing the Argo CD instance allows administrators to separate applications that require elevated privileges from other applications meant to manage other resources.
  1. Making the repository private.
  2. Enabling branch protection on the main development branch of the CI pipeline.
  3. Requiring signed commits.
  4. Requiring code reviews before merging branches.
  5. Restricting who can push to the main development branch.
  6. Adding automated processing to lint pull requests, labeling the pull request as sensitive if it contains changes that affect access control, such as roles and role bindings.

Lesson #2: Argo resources are for Argo admins only

  1. Argo CD only recognizes Application resources in the namespace hosting the Argo CD instance (see discussion in this issue.)
  2. From the previous point, a Project containing Application resources must include the namespace of the Argo CD instance in its destinations field (see this other discussion.)
  3. Since that destinations field applies to the contents of all repositories in the project, only Argo CD admins or delegates can merge pull requests in those repositories.

Lesson #3: Delete the “default” project

Diagram with a big box labeled “Project default”, referencing multiple repositories with a bomb icon next to them and a sticky figure with a mask and in a suspicious manner. The project also has an arrow pointing at destinations, with some of them containing the same bomb icon seen in the source repositories.
The “default” application project in Argo CD allows applications from all sources to manage resources on all destinations. It is meant solely for learning purposes in a disposable environment.

Lesson #4: Block ClusterRoleBindings in (most) projects

Lesson #5: Narrow roles on remote clusters

  1. Determine the list of namespaces in the cluster that the Kubernetes administrators want to manage with Argo CD applications. Is it the whole cluster, or is it specific namespaces?
  2. Ask the Kubernetes administrator to create a dedicated service account in each cluster, such as “argocd-control-plane-account.”
  3. Ask the Kubernetes administrator to bind that service account to a role or roles) that match the specific namespaces discussed in the previous steps. Full permissions in the entire namespace may be acceptable. Still, one may consider narrower permissions depending on the scenario (for instance, not granting permissions to read secret resources.)
  4. Ask the Kubernetes administrator to generate the “kubeconfig” file from that service account, then use that file when adding the cluster to Argo CD.
  5. Agree on the rotation interval for the service account tokens and the process to replace them in Argo.
Two-part figure. On the left, a Kubernetes administrator stares at 3 boxes, representing clusters. Each box has a hole with a different shape, representing portions of the the cluster that will be managed with GitOps. On the right, the Kubernetes administrator carries a stack of papers representing credentials for those three portions, with “90 days” written on top of the stack. A sitting Argo CD administrator uses those instructions to type commands starting with “argocd cluster add”.
Cluster administrators should create dedicated service accounts with the narrowest possible roles before handing “kubeconfig” files to Argo CD administrators.

Lesson #6: Have a CVE response plan ready

Left side of figure has sticky figure with a megaphone announces a new CVE. Right side of figure has administrators frantically trying to apply mitigations and patches.
Once a CVE is announced, administrators should be ready to react quickly with patches and mitigations.

Summary

--

--

--

Operations architect, corporate observer, software engineer, inventor. @dnastacio

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Object-Oriented Programming in Java

Testing Sockets in an Echo Server

A full-featured multi-tenant app with Laravel Part 4 —Tenancy aware Authentication.

Large Things Living in Cold Places

Case Study of DCS Upgrade: How to Reduce Stress During Execution

Ways to add new columns to a Python Pandas DataFrame

Learning Go with Leetcode (Part 1 of .)

Test Post

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Denilson N.

Denilson N.

Operations architect, corporate observer, software engineer, inventor. @dnastacio

More from Medium

A GitOps-Powered Kubernetes Testing Machine with ArgoCD and Testkube — Kubeshop

GitOps Nirvana: controlled and safe promotions of changes across environments with Terraform.

Effective CI/CD strategies for updating GitOps repositories

Person placing box on long conveyor belt rolling to the right, with pile of processed boxes to the right of the conveyor belt.

Measuring Argo Workflow Costs with Kubecost