OpenShift GitOps with ArgoCD Vault Sidecar

This repository is intended to be a refreshed guide for deploying OpenShift GitOps with the ArgoCD Vault Plugin to retrieve secrets from a Hashicorp Vault instance and inject them into Kubernetes resources.

This guide builds on previous work like this Red Hat blog from 2021 and is updated to use the more modern custom plugin sidecar approach.

Pre-requisites

This guide assumes you have a running OpenShift 4.12+ cluster with administrative privileges on that cluster. In my case I will be running a Red Hat OpenShift 4.15 cluster provisioned through the Red Hat demo system.

Additionally, ensure you are logged into the cluster in a terminal environment with oc and helm binaries available before proceeding with any of the following steps:

# Check pre-requisites
oc version && helm version

# Login to cluster
oc login --token=<token> --server=<server api url>:6443

Step 1 - Deploy vault

Our first step is to deploy an instance of vault, note that this example is deploying vault in dev mode which is not suitable for production. If you already have a vault instance running you can skip this step.

helm upgrade --install vault vault \
    --repo https://helm.releases.hashicorp.com \
    --namespace vault \
    --create-namespace \
    --set "global.openshift=true" \
    --set "server.dev.enabled=true" \
    --set "injector.enabled=false"

Step 2 - Configure vault

Once we have a vault instance deployed we can run a shell script against the vault pod to quickly configure it for this demo using the excellent vault cli.

The configuration script will configure the kubernetes auth method for vault, enable a kv-v2 secret store, create a test secret and an access policy for that secret.

# Copy our config shell script to the vault pod
oc --namespace vault cp 1-vault/configure-vault.sh vault-0:/tmp/configure-vault.sh

# Run the script remotely in the vault pod
oc --namespace vault exec vault-0 -t -- sh -c 'cd /tmp && ./configure-vault.sh'

Step 3 - Configure argocd vault plugin auth to vault

Once vault is deployed and configured we need to do some further configuration to enable our soon to be deployed ArgoCD Vault Plugin to be able to authenticate to Vault using a kubernetes service account.

# Create namespace that we will deploy argocd into
oc new-project vplugindemo

# Create the service account to be used by argo vault plugin to auth to vault
oc create serviceaccount vplugin

# Create a role in vault to bind our service account to the policy we created earlier
oc --namespace vault exec vault-0 -- vault write auth/kubernetes/role/vplugin \
    bound_service_account_names=vplugin \
    bound_service_account_namespaces=vplugindemo \
    policies=vplugin \
    ttl=1h

# Create the secret for the argo vault plugin to use to configure vault connection
# Supported parameters list: https://argocd-vault-plugin.readthedocs.io/en/stable/config/
oc --namespace vplugindemo create -f 2-argocd/secret-vault-configuration.yaml

Step 4 - Configure argocd vault plugin processing

With authentication configured, we now need to configure what our ArgoCD Vault Plugin sidecar will actually do. This is a two step process, firstly specifying a discover command, then a following generate command.

Refer to these documented examples including for helm or kustomize based applications. In our example we will take the most basic approach of discovering files that contain an annotation, then using argocd-vault-plugin generate . to template the files.

oc --namespace vplugindemo create -f 2-argocd/configmap-plugin.yaml

Step 5 - Deploy openshift gitops

With vault deployed, configured and our argocd vault plugin configured, let’s now deploy the OpenShift GitOps operator.

Note: The below operator subscription is pinned specifically to v1.12.3 from the gitops-1.12 release channel.

oc --namespace openshift-gitops-operator create -f 2-argocd/namespace-gitops.yaml
oc --namespace openshift-gitops-operator create -f 2-argocd/operatorgroup-gitops.yaml
oc --namespace openshift-gitops-operator create -f 2-argocd/subscription-gitops.yaml

Once the operator has installed successfully we can create our argocd instance via custom resource.

oc --namespace vplugindemo create -f 2-argocd/crd-argocd.yaml

Step 6 - Create argocd application

Once argo is running, create this baseline sample application. This will create a secret resource on the cluster that will have the default placeholder values replaced with values that come from vault! 🎉

oc --namespace vplugindemo create -f 2-argocd/application-example.yaml

Once the application has been created it should automatically sync. We can check the contents of the secret as follows:

oc get secret example-secret -o jsonpath={.data.username} | base64 --decode
oc get secret example-secret -o jsonpath={.data.password} | base64 --decode