How To Access Vault Secrets Inside of Kubernetes Using External Secrets Operator (ESO)

The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

In Kubernetes, you can use secrets in pods to avoid keeping connection strings and other sensitive data in source control or to prevent your application from accessing sensitive data directly. Storing secrets in a secret store such as HashiCorp Vault is a secure way to allow access for the required tools. Secret stores often provide features such as HTTP APIs to interact with them securely. Tools like Vault also usually provide ways to integrate with Kubernetes, such as by using sidecars. However, one downside of this approach and other ways these tools integrate with Kubernetes is that they are usually application-specific and create a tight coupling to the tool, which you might wish to avoid so that you can easily switch the secrets provider if needed.

External Secrets Operator (ESO) is a Kubernetes operator that adds several Custom Resource Definitions (CRDs) and allows you to consume the native Kubernetes secrets resource. It does this by creating Kubernetes secrets and keeping them in sync with a specified secret located in a secrets provider, whether that be HashiCorp Vault, AWS Secrets Manager, GCP Secrets Manager, or any other supported provider. The main benefit of this system is that by referencing native Kubernetes secrets, any services consuming secrets in Kubernetes are decoupled from these tools. This separation allows you to more easily change the backing provider without needing to change anything else.

In this tutorial, you will install the External Secrets Operator in a DigitalOcean-managed Kubernetes cluster and set up a SecretStore and an ExternalSecrets to fetch values from a HashiCorp Vault instance.

At the end, you will have a working setup in which you can fetch secrets from any provider supported by the External Secrets Operator to be used inside of a Kubernetes cluster.

As with any service that manages sensitive information, you should consider reading additional documentation regarding Vault’s deployment best practices before using it in a production-like environment. For example, Vault’s production hardening guide covers topics such as policies, root tokens, and auditing.

If you’re looking for a managed Kubernetes hosting service, check out our simple, managed Kubernetes service built for growth.

Prerequisites

To complete this tutorial, you will need:

• A DigitalOcean account. If you do not have one, sign up for a new account.

• A Kubernetes Cluster set up using the DigitalOcean Control Panel. To get started, see the DigitalOcean Kubernetes guide for creating clusters.

  • Other methods of setting up clusters, such as the tutorial How To Create a Kubernetes Cluster Using Kubeadm on Ubuntu 20.04, work as well, but this tutorial uses the Control Panel method.

• Doctl installed, which you can do by following our guide on How to Install and Configure doctl.

• The kubectl command-line tool installed on your local machine and configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

• Helm installed. For installation, see Step 1 of our tutorial, How To Install Software on Kubernetes Clusters with the Helm 3 Package Manager.

• Vault installed on your machine, with a Secret you have created. To install Vault and securely create a Secret, follow our tutorial, How To Securely Manage Secrets with HashiCorp Vault on Ubuntu 20.04. Save the access tokens created in this tutorial to use in a later step.

  • You can use any other provider that is supported by the External Secrets Operator, but this tutorial will use HashiCorp Vault. You can find a list of providers in the Vault product documentation.

Step 1 — Installing the External Secrets Operator (ESO)

In this step, you will install the External Secrets Operator via Helm into your Kubernetes cluster. Since you will be using Helm to install the External Secrets Operator, you’ll need to make sure that your Kubernetes context is set up correctly.

To set this up, start by downloading your Kubeconfig file from the DigitalOcean Cloud Control Panel of your Kubernetes Cluster. On the Overview tab, locate the Configuration section, and select the Download Config File button. The file is named <clustername>-kubeconfig.yaml. You can put this file anywhere, but the suggested location is the ~/.kube folder. Then, set your environment variable KUBECONFIG equal to the path of the file using one of the following commands.

On Linux or Mac, you can run the command:

export KUBECONFIG=/home/sammy/.kube/clustername-kubeconfig.yaml

On Windows from Powershell, you can use the command:

 $env:KUBECONFIG = "C:\Users\sammy\.kube\clustername-kubeconfig.yaml"

Next, confirm your current context by running the following command:

kubectl config current-context

Your output will look similar to the following:

my-k8s-cluster

Once your current context is set up, you can then add the external-secrets repo to Helm:

helm repo add external-secrets https://charts.external-secrets.io

The output will look like the following:

"external-secrets" has been added to your repositories

Next, make sure you have the most up-to-date repo:

helm repo update

Your output will be similar to the following:

...Successfully got an update from the "external-secrets" chart repository

Once the repo has been added, you can use Helm to install the external-secrets repository you added:

helm install external-secrets \
    external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

With the -n external-secrets option, you’re specifying that the external-secrets repository should be installed in the external-secrets namespace. With --create-namespace, the namespace will be created it if it doesn’t exist. Lastly, you set --set installCRDS to true to make sure that the Custom Resource Definitions for External Secrets Operator are installed with it.

Your output will look like the following:

NAME: external-secrets
LAST DEPLOYED: {Date goes here}
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully!

In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets

In this step, you installed the External Secrets Operator via Helm into your Kubernetes cluster, which will allow you to create a SecretStore and an ExternalSecret. In the next step, you’ll create a SecretStore.

Step 2 — Creating a Secret Store

In this step, you will create a SecretStore, which is what External Secrets Operator uses to store information about how to communicate with the given secrets provider. But before you work with the External Secrets Operator, you’ll need to add your Vault token inside Kubernetes so that the External Secrets Operator can communicate with the secrets provider.

As part of the prerequisites, you may have followed the tutorial, How To Securely Manage Secrets with HashiCorp Vault on Ubuntu 20.04, which generated a Vault token to use for authentication. In this step, you can use either the root token you got from unsealing your Vault (not recommended for production) or the token from Step 5 of that tutorial, which is about creating an Authorization Policy in Vault.

You are going to put this token inside of Kubernetes as a secret so that External Secrets Operator can use it to communicate with Vault. You can do this by running the following command, replacing the highlighted portion with your Vault token:

kubectl create secret generic vault-token --from-literal=token=YOUR_VAULT_TOKEN

Your output will look like the following:

secret/vault-token created

You can verify this by running the following command, which gets all secrets in the default namespace (which is where yours will be unless you added a namespace to the previous command):

kubectl get secrets

The output will look something like the following:

NAME                  TYPE                                  DATA      AGE
vault-token           Opaque                                1         2m

With this secret created, you can start working with the External Secrets Operator. The first step is setting up a SecretStore, which holds the information for contacting a secret provider and identifies which provider to use. Secret stores can be accessed by any external secret in the same namespace, so be sure to segment your namespaces as needed. In this tutorial, you will put everything in the default namespace, but this is not recommended for a production cluster.

Using nano or your favorite text editor, create a secret-store.yaml file and paste the following contents into it. Be sure to fill in the data that is specific to your setup, such as the server field, which is the endpoint where your Vault server can be accessed:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://your-domain:8200"
      path: "kv"
      version: "v1"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"

Under the provider field, you specify vault as the provider as well as where to reach it using the server field. You also specify the path kv, which is the root path where your secrets are located. If you followed the Vault prerequisite tutorial but named your SecretStore something other than kv, then be sure to change that field accordingly.

v1 of the KV engine should be enabled by default. If you changed the version of the kv engine in Vault when enabling it, you can also specify the version as v2.

The final field is the auth section, where you specify the secret that you created before by providing the name of the secret and the key inside of the secret that the token lives in. To learn more about other Authentication methods for Vault, check out the product documentation.

If you’re interested in learning more about other configuration options that are available for SecretStores, you can find the CRD specification in the External Secrets product documentation, which contains all of the fields that can be provided, and some other fields added by Kubernetes that can be fetched after it’s added.

Save and close the file.

Create the SecretStore by running the following command:

kubectl apply -f ./secret-store.yaml

This command applies this CRD to your cluster and creates the object. You can see the object by running the following command, which will show you all of the information about the object inside of Kubernetes:

kubectl get SecretStore vault-backend -o yaml

Your output will look similar to the following:

apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  annotations:
    # Some kubernetes annotations
  creationTimestamp: # Timestamp of the creation time
  generation: 1
  name: vault-backend
  namespace: default
  resourceVersion: "12345"
  uid: # a UID for the resource
spec:
  provider:
    vault:
      auth:
        tokenSecretRef:
          key: token
          name: vault-token
          namespace: default
      path: kv
      server: https://your-domain:8200
      version: v1
status:
  conditions:
    - type: Ready
      status: "False"
      reason: "ConfigError"
      message: "SecretStore validation failed"
      lastTransitionTime: "0000-00-00T00:00:00Z"

If you created the SecretStore successfully, you won’t see the status field, but if you see a type: Ready and a status: "False", then the reason and message fields will provide more information about what is wrong. A very common issue is could not create client. This generally means that the authentication method for your client has failed as the SecretStore will try and create a client for your provider to verify everything is working. With the authentication method of token for this tutorial, be sure to check that your SecretStore resource has the correct namespace for your secret and that the name and key match exactly with what you created.

Troubleshooting

Depending on how Vault was set up, you might encounter some errors when creating a SecretStore. One of the most common is message: unable to validate store. This could be several things, but there are a few things you can check, especially if you used the prerequisite tutorial to install Vault and securely create a secret.

First, you’re going to tell Vault to listen to all incoming requests so that you can remotely access it by changing the Vault configuration file. If you followed the prerequisite Vault tutorial, then you’ll change the /etc/vault.d/vault.hcl file.

Using nano or your favorite text editor, open /etc/vault.d/vault.hcl for editing. Inside the configuration file under the listener "tcp" section, check the address. If the address is 127.0.0.1:8200, change it to 0.0.0.0:8200 as shown:

...
listener "tcp" {
  address       = "0.0.0.0:8200"
...
}

This change tells Vault to listen for external connections so you can reach it from the External Secrets Operator. Restart Vault (be sure to unseal it again), and it should now accept external connections.

A second issue might be the firewall blocking connections on port 8200, which can be fixed by updating the firewall:

ufw allow 8200

In this section, you created a secret in Kubernetes to hold your information on how to authenticate with your secrets provider. Then you created a SecretStore resource that tells the External Secrets Operator how to contact your provider. In the next step, you’ll create an ExternalSecret.

Step 3 — Creating an External Secret

In this step, you will create an ExternalSecret, which is the main resource in the External Secrets Operator. The ExternalSecret resource tells ESO to fetch a specific secret from a specific SecretStore and where to put the information. This resource is very important because it defines what secret you’d like to get from the external secret provider, where to put it, which secret store to use, and how often to sync the secret, among several other options.

To create an ExternalSecret, begin by creating a file called external-secret.yaml and add the following code:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-cool-secret
spec:
  refreshInterval: "15s" # How often this secret is synchronized
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target: # Our target Kubernetes Secret
    name: my-cool-new-secret # If not present, then the secretKey field under data will be used
    creationPolicy: Owner # This will create the secret if it doesn't exist
  data:
    - secretKey: my-cool-new-secret-key
      remoteRef:
        key: message # This is the remote key in the secret provider (might change in meaning based on your provider)
        property: value # The property inside of the secret inside your secret provider

Here, you make an ExternalSecret that references your SecretStore from the previous step. Under the target field, you’re telling the External Secrets Operator the information about the Kubernetes secret you’d like to create. The data section is where your secret lives inside of your secret provider (Vault, in this instance), and contains the following properties:

• secretKey: This is the key inside of the Kubernetes secret that you would like to populate.

• remoteRef.key: This is the key inside of the secret provider. For example, in Vault, you can have secrets located at different paths. Here, you could use something like myfolder/coolsecrets/mysecret as the value.

• remoteRef.property: This is the property inside of the secret at the path specified in remoteRef.key that you would like to put inside of the key specified in secretKey.

The data field is an array in the YAML, so you can specify multiple secrets that you would like to put into a single Kubernetes secret. External Secrets Operator also offers some advanced template logic that you can use to build things like connection strings, which you can read more about in the Advanced Templating section of the production documentation. You can also read about all of the fields and options around ExternalSecrets by going through the API external secret reference docs.

Save and close your file when you’re done.

Next, create this resource with the following command:

kubectl apply -f ./external-secret.yaml

After it has been applied, you can then make sure that all is well by running the following command:

kubectl get ExternalSecret my-cool-secret

Your output will look similar to the following:

NAME             STORE           REFRESH INTERVAL   STATUS
my-cool-secret   vault-backend   15s                SecretSynced

The output confirms the successful creation of the ExternalSecret resource.

Troubleshooting

If the previous output has a Sync Error under STATUS, then be sure that your SecretStore is set up correctly. You can view the actual error by running the following command:

kubectl get ExternalSecret my-cool-secret -o yaml

Your output will be similar to the following, which will give more information about the error:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-cool-secret
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: my-cool-new-secret
    creationPolicy: Owner
  data:
    - secretKey: my-cool-new-secret-key
      remoteRef:
        key: test-secret
        property: my-cool
status:
  # refreshTime is the time and date the external secret was fetched and
  # the target secret updated
  refreshTime: "2019-08-12T12:33:02Z"
  # Standard condition schema
  conditions:
    # ExternalSecret ready condition indicates the secret is ready for use.
    # This is defined as:
    # - The target secret exists
    # - The target secret has been refreshed within the last refreshInterval
    # - The target secret content is up-to-date based on any target templates
    - type: Ready
      status: "True" # False if last refresh was not successful
      reason: "SecretSynced"
      message: "Secret was synced"
      lastTransitionTime: "2019-08-12T12:33:02Z"

In this output, you’ll see status fields, and maybe some extra annotations (which you can ignore, as they’re usually metadata about the resource). In the status section, you can see information about whether the ExternalSecret has been properly synced or if something has gone wrong.

In this step, you created an ExternalSecret resource. The ExternalSecret resource allows you to specify values in your secrets provider that you’d like to fetch as well as where to put them in Kubernetes. For more advanced features, you can check out the External Secrets product documentation.

Conclusion

Now that you have successfully set up External Secrets Operator into your Kubernetes cluster, and also deployed a SecretStore and ExternalSecret, you can deploy any other secrets that you’d like.

External Secrets Operator also has many other features and Custom Resources that allow you to set up your Kubernetes cluster in the way that best suits your needs. Additionally, HashiCorp Vault is not the only provider, and setup is similar to what was described in this tutorial. You can learn more from the External Secrets documentation.

This approach to secrets in Kubernetes is powerful and extendable, as the operator is open source, and allows you to use various providers for secrets without having to worry about the specific providers in anything consuming the secret.

Finally, there are other resources to explore. One is the ClusterExternalSecret, which is new in version 0.5.0 of the External Secrets Operator. This resource allows you to define an ExternalSecret and then have the operator create this ExternalSecret in every namespace that matches a selector you define. You can read more on this feature in the product documentation. Another resource is the ClusterSecretStore, which allows an ExternalSecret from any namespace to access it (whereas a normal SecretStore can only be accessed in the same namespace where it lives).