Six Critical Blindspots While Securing Argo CD

8 min readApr 21, 2022

Lately, I have been addressing security risks in our local GitOps deployments, specifically around Argo CD* control planes.

GitOps frameworks like Argo CD run with elevated privileges relative to target deployments, making them prime targets for supply chain attacks.

Keeping systems secure is a continuous effort over time. This article shows the core strategies for securing an Argo CD deployment and keeping you ahead of potential exposures. If you are new to Argo CD, you may want to read it side-by-side with the Argo CD security manual.

System administrator sits at expansive network control center wearing headphones and listening to music. A hacker crouches in a corner typing commands. A robot carries out the commands on a deployment environment without catching the attention of the system administrator. — GitOps control planes have elevated privileges on entire deployment environments, making them a prime target for supply chain attacks.

Applications and Projects

Before diving into risk analysis, it is helpful to briefly mention Argo CD’s basic deployment unit: Applications.

An Argo CD Application combines a source location (Git URL, revision, and folder containing resources) and a target destination (a namespace in a Kubernetes cluster.)

Authorized users add applications to an Argo CD Project. An administrator configures a project to restrict the range of sources and destinations in applications and creates access control lists of people authorized to work with the applications in the project.

In addition to these capabilities, I also like that one can issue scoped tokens from a project and use them on external CIs, much like GitHub access tokens.

Diagram with big “Project X” box at the center and arrows leading to the concepts contained in an Argo CD project: source repository, whitelisted resources that can be present in Applications in that project, a group of people who can manage the project, and finally, a block of clusters that act as destinations. — An Argo CD project combines user permissions, lists of allowed resources, source repositories, and destination clusters.

Lesson #1: Use a dedicated project for the control plane

This lesson assumes that you manage the Argo CD control plane through GitOps — how else, right?

Why it matters: Without a dedicated project for the control plane, a project shared with other applications allows those applications to manage workloads inside the Argo CD namespace.

The corresponding application resource(s) for the control plane must target the location of the control plane squarely with the following fields:

spec.destination.server refers to the local cluster: https://kubernetes.default.svc
spec.destination.namespace refers to the namespace where the Argo CD instance is running.

The dedicated project should be configured with the following settings:

spec.sourceRepos[] refers to the Git repositories managed by the instance owners and only those repositories containing the control plane’s configuration.
spec.destinations[0].server refers to the local cluster: https://kubernetes.default.svc
spec.destinations[0].namespace refers to the namespace running the Argo CD instance

A diagram with a “Project control-plane” at the center, with an arrow to a “control-gitops” GitOps repository, a reference to the local cluster, and an arrow to a sticky figure sitting at a desk and typing configuration files, representing an Argo administrator. — A dedicated project for managing the Argo CD instance allows administrators to separate applications that require elevated privileges from other applications meant to manage other resources.

No other project in the instance should have that same combination of server and namespace in its destinations field.

Note that the attack surface of the control plane extends to the Git repository hosting the control-plane resources. In that sense, Argo CD administrators must also be well-versed in managing and securing Git repositories.

Securing a GitOps repository is a topic that deserves its own future article. For now, I suggest a minimum of:

Making the repository private.
Enabling branch protection on the main development branch of the CI pipeline.
Requiring signed commits.
Requiring code reviews before merging branches.
Restricting who can push to the main development branch.
Adding automated processing to lint pull requests, labeling the pull request as sensitive if it contains changes that affect access control, such as roles and role bindings.

Lesson #2: Argo resources are for Argo admins only

Only GitOps repositories owned by Argo CD administrators should contain resources belonging to the argoproj.io API group.

Why it matters: Popular Argo tutorials contain resources like ApplicationSet.argoproj.io and Application.argoproj.io. These resources require their projects to allow the creation of other resources in the same namespace as the Argo CD instance.

Multiple users consider this lesson quite restrictive, but that is the current security model. The chain of explanations goes as follows:

Argo CD only recognizes Application resources in the namespace hosting the Argo CD instance (see discussion in this issue.)
From the previous point, a Project containing Application resources must include the namespace of the Argo CD instance in its destinations field (see this other discussion.)
Since that destinations field applies to the contents of all repositories in the project, only Argo CD admins or delegates can merge pull requests in those repositories.

You can read the detailed discussion in this issue, which led to proposals calling for further attention on both documentation and the user interface.

Beyond calling attention to this common security blind spot, there are proposals to tighten the security privileges of an Application within an instance, such as issue 7689.

Lesson #3: Delete the “default” project

Argo CD’s initial configuration has a “default” project. This project allows authorized users to add applications from any source repository and manage all resources on all destinations.

Why it matters: A malicious or poorly designed Application may contain resources that alter the configuration of the entire cluster. The service accounts used to run Argo CD are often privileged enough to oblige.

That configuration is ideal for a non-production Argo CD instance tuned for fast progress during a learning session. It is also the stuff of nightmares for an unsuspecting system administrator.

Diagram with a big box labeled “Project default”, referencing multiple repositories with a bomb icon next to them and a sticky figure with a mask and in a suspicious manner. The project also has an arrow pointing at destinations, with some of them containing the same bomb icon seen in the source repositories. — The “default” application project in Argo CD allows applications from all sources to manage resources on all destinations. It is meant solely for learning purposes in a disposable environment.

A cluster may still reject requests due to its own RBAC permissions, but removing broad permissions is an essential first line of defense for the control plane.

Note #1: The Argo CD UI prevents the deletion of the project, so delete the project directly using kubectl delete AppProject default -n ${argocd_namespace:?}.

Note #2: (added on 06/06/2022) Matthew Hembree called out in the comments section that although one could modify the settings of the “default” project to be more restrictive, it is still better to delete the project because an empty spec.project field in an Application translates to the “default” project. One could argue that defaulting to that project may be the desired behavior for some applications, but it may also blindside administrators, so it is best to push administrators into making conscious decisions about Application privileges.

Note #3: (added on 10/26/2022) Michael Crenshaw opened a corresponding issue in Argo CD to narrow down some of the permissions in the “default” project.

Lesson #4: Block ClusterRoleBindings in (most) projects

The previous lessons prevent unauthorized repository owners from commandeering an Argo CD instance. This lesson is about protecting the cluster itself.

Why it matters: Applications managing ClusterRoleBindings can attach privileged cluster roles to a service account and then use that account to execute privileged actions from pods running resource hooks.

The only potential exception to this rule is when a dedicated Argo CD project is assigned to the cluster administrators.

While we all have dealt with our share of products requiring admin-level privileges during their installation, blocking these resources at the project level forces meaningful conversations between product and cluster admins.

These conversations can lead to safer arrangements, with narrower roles tied to separate service accounts and managed in git repositories owned by cluster administrators.

Lesson #5: Narrow roles on remote clusters

Argo CD offers distinct deployment topologies where an instance can control anything from the local cluster to a fleet of remote clusters.

Why it matters: If the Argo CD control plane is compromised, narrower permissions assigned to a dedicated service account in the remote cluster will limit the exposure and speed up recovery efforts.

When adding remote clusters to the list of destinations in Argo CD, I suggest spending a few minutes discussing the following points with the cluster administrators:

Determine the list of namespaces in the cluster that the Kubernetes administrators want to manage with Argo CD applications. Is it the whole cluster, or is it specific namespaces?
Ask the Kubernetes administrator to create a dedicated service account in each cluster, such as “argocd-control-plane-account.”
Ask the Kubernetes administrator to bind that service account to a role or roles) that match the specific namespaces discussed in the previous steps. Full permissions in the entire namespace may be acceptable. Still, one may consider narrower permissions depending on the scenario (for instance, not granting permissions to read secret resources.)
Ask the Kubernetes administrator to generate the “kubeconfig” file from that service account, then use that file when adding the cluster to Argo CD.
Agree on the rotation interval for the service account tokens and the process to replace them in Argo.

Two-part figure. On the left, a Kubernetes administrator stares at 3 boxes, representing clusters. Each box has a hole with a different shape, representing portions of the the cluster that will be managed with GitOps. On the right, the Kubernetes administrator carries a stack of papers representing credentials for those three portions, with “90 days” written on top of the stack. A sitting Argo CD administrator uses those instructions to type commands starting with “argocd cluster add”. — Cluster administrators should create dedicated service accounts with the narrowest possible roles before handing “kubeconfig” files to Argo CD administrators.

Lesson #6: Have a CVE response plan ready

Like any software component, GitOps frameworks may suffer from vulnerability and exposures, and mitigating those risks requires awareness and speed.

Why it matters: Having a response plan in place expedites the assessment of the situation and eventual recovery efforts.

ArgoCD publishes its CVEs as GitHub security advisories, making it easier to stay on top of new developments. It is also good to familiarize yourself with Argo CD issues labeled with the “security” tag to stay ahead of new developments and help shape their progress with comments and suggestions.

As a consumer of these frameworks, there is no fixed recipe for responding to such advisories. The best strategy is to read through sections related to workarounds and mitigations and implement those recommendations as quickly as possible.

Left side of figure has sticky figure with a megaphone announces a new CVE. Right side of figure has administrators frantically trying to apply mitigations and patches. — Once a CVE is announced, administrators should be ready to react quickly with patches and mitigations.

Patching a CVE is not sufficient. The next step is to determine whether someone may have exploited the CVE. The assessment, at a minimum, should involve these steps:

Inspect Argo-generated audit events and logs of calls related to applications used to manage the control plane.
Revisit all roles and role bindings in the cluster hosting the control plane.

Summary

Argo CD’s out-of-the-box permissions are tuned for a quick on-ramp into learning exercises. They are unsuited for production environments and compounded by many early adopters not realizing the security implications of using the popular “App of Apps” pattern.

Use separate Argo instances for individual teams, create a dedicated project for managing Argo, and remember to delete the “default” project.

When separate teams manage remote clusters, discuss service accounts’ roles and token rotation policies with the Kubernetes administrators before registering the clusters as destinations in Argo.

And finally, be prepared to monitor and react to Argo vulnerabilities with validated plans for mitigating the risks outlined in a CVE and post-incident assessments about eventual exploits in your systems.