Teams at GoDaddy use the AWS managed Kubernetes offering, EKS, to deploy their services. We also use AWS Secrets Manager for storing secrets, like private keys and database passwords. EKS, however, does not provide much support for accessing Secrets Manager. Therefore, teams develop custom solutions for accessing secret data from their EKS clusters. One approach, for example, is to fetch secrets from Secret Manager when the application starts. The result is overlapping efforts for developing these custom solutions and because security is critical in this context, the individual efforts can require substantial engineering time.
This blog post describes a generalized approach and implementation for supporting secret management systems, like AWS Secrets Manager, in Kubernetes. We call this system Kubernetes External Secrets and we have open sourced our initial implementation. Our current solution focuses on making it easy for developers to manage secrets with AWS and deploy them to their EKS clusters, but we think our approach is general enough for other types of Kubernetes clusters and secret management systems.
The rest of this blog post describes the motivation for and design of Kubernetes External Secrets. The README.md has instructions for adding Kubernetes External Secrets to your cluster if you are looking to get started immediately.
Kubernetes has a built-in object for managing secrets called a
Secret object is convenient to use: it provides a declarative API
that makes it easy for application
to access secret data without any special code. One downside of
Secret objects is that they do not support storing or retrieving
secret data from external secret management systems. It’s often
beneficial, however, to use Kubernetes with an external service that
handles secret management and includes useful features. For
example, Secrets Manager integrates with other AWS services, like
Lambda functions, includes encryption at rest, and has a useful
mechanism for codifying rotation policies.
Kubernetes External Secrets aims to provide the same ease of use as
Secret objects and provide access to secrets stored
externally. It does this by adding an
ExternalSecret object to the
Kubernetes API that allows developers to inject external secrets into
Pod using a declarative API similar to the native
Instead of inlining base64 encoded secret data into a
developers define an
ExternalSecret object that specifies a secret
management system and properties to load from that system.
For example, instead of using a
Secret object to store a database
apiVersion: v1 kind: Secret metadata: name: cats-and-dogs type: Opaque data: password: d29vZgo=
a developer can use an
ExternalSecret, specifying the secret
management systems as
backendType and the properties to access in the
apiVersion: 'kubernetes-client.io/v1' kind: ExtrenalSecret metadata: name: cats-and-dogs secretDescriptor: backendType: secretsManager data: - key: cats-and-dogs/mysql-password name: password
and access that password from a
Pod in the same way they would
if they were using a
apiVersion: v1 kind: Pod metadata: name: cats-and-dogs spec: containers: - name: cats-and-dogs image: cats-and-dogs env: - name: SECRET_PASSWORD valueFrom: secretKeyRef: name: cats-and-dogs key: password
Notice that the
ExternalSecret does not contain secret data. It’s
safe to store in plain text along with your other Kubernetes manifest
files for your service.
Kubernetes External Secrets adds the
ExternalSecret object to
Kubernetes using a
and adds an
ExternalSecret controller we wrote in Node.js that implements
the behavior of the object type itself.
ExternalSecret controller follows a familiar pattern seen in
other Kubernetes objects, like
users declare a desired state in an
ExternalSecret object and the
controller creates or updates a complementary
Secret object to reach
that state. The controller monitors
ExternalSecret objects, fetches
secret data from the specified external secret management system, and automatically
Secret objects that hold the secret data. The
architecture diagram below illustrates this process.
You can add Kubernetes External Secrets to your cluster with a single command:
kubectl apply -f https://raw.githubusercontent.com/godaddy/kubernetes-external-secrets/master/external-secrets.yml
and begin manipulating
ExternalSecret objects like you would any
other Kubernetes object type:
kubectl -n cats-and-dogs get es
If you’re interested in converting existing
Secret objects to
ExternalSecret objects, one of the authors of Kubernetes External
Secrets wrote a Kubernetes External Secrets
makes migration easy by converting a
Secret to a series of AWS
CLI commands and a kubectl command that loads your secret data
into AWS Secrets Manager and creates a complementary
We experimented with and drew inspiration from other projects that help manage access to secret data from Kubernetes.
cmattoon/aws-ssm uses annotations on
Secret objects to identify properties in AWS Parameter Store and
Secret object with those properties’ values.
kubesec makes it easy to encrypt
data before storing it in a
Secret object or manifest. At run-time,
applications are responsible for decrypting the data before using it.
The Kubernetes Vault
provides Vault auth tokens to
Pod objects. Application code running
Pod can use that token to authenticate with Vault and retrieve
We explored configuring nodes in our CICD pipeline to inject
Secret objects during testing and deployment. One benefit of this
approach is that Kubernetes clusters would not need to access the
external secret management system directly, which might help mitigate
data leaks during an attack. On the other
hand, it required the Kubernetes manifest files for a single
application to be stored in several locations and complicated
application debugging and deployment because developers would need to
use a combination of kubectl and specialized CICD tools.
Kubernetes External Secrets supports AWS Secrets Manager and AWS Systems Manager, however, we believe the approach is general enough to support other external secret management systems and would be excited about working with community members to add them.
The Kubernetes External Secrets controller does not have a broad attack surface since it is accessible only via the Kubernetes API. We have worked, however, to tighten security by leveraging RBAC to restrict what the controller can access, using eslint-plugin-security to identify potentially vulnerable code patterns, and doing design reviews at GoDaddy. Nevertheless, we think it is important to continually improve security and take advantage of new features and patterns as they emerge. We plan on continuing to improve security by:
Pods, that will help harden our implementation.
We hope that Kubernetes External Secrets can provide teams outside of
GoDaddy with a common way to access secret data in secret management
systems. Using Kubernetes External Secrets does not require any
application modifications and existing
Pod specifications that
Secret objects will work with
without any changes.
If you want to get started with Kubernetes External Secrets or contribute please visit https://github.com/godaddy/kubernetes-external-secrets.