How to Use AWS Secrets Manager with Kubernetes (EKS)

The default pattern for secrets in Kubernetes — environment variables from ConfigMaps or opaque Secrets baked into manifests — puts sensitive values in etcd and in any CI system that touches the cluster. For EKS workloads, the better path is AWS Secrets Manager backed by IRSA (IAM Roles for Service Accounts): secrets live in a managed store, access is scoped per service account, and rotation can be handled independently of the application.

This covers the full integration: creating the secret, wiring up OIDC, scoping the IAM trust relationship, deploying the Secrets Store CSI Driver, and consuming the secret both as a mounted file and as an environment variable.

Prerequisites

• AWS CLI configured with an admin IAM user (programmatic access)
• eksctl and kubectl installed
• Basic familiarity with Kubernetes concepts

Goals

• Secret stored in AWS Secrets Manager and accessible from inside a pod
• IAM authentication wired up via OIDC and a service-account-scoped trust policy
• Secrets Store CSI Driver and AWS provider running in the cluster
• Deployment that mounts the secret as a file and exposes it as an environment variable

Step 1 — Create a Secret in AWS Secrets Manager

Head to the AWS Secrets Manager console and create a new secret:

1. Choose Other type of secret and add a key-value pair (e.g., my_api_token → <your-value>).
2. Give the secret a descriptive name such as prod/service-token. You'll reference this name later in Kubernetes.
3. Skip Resource Permissions and the rotation section for now — access will be granted directly via an IAM role.

Once created, open the secret and copy its full ARN (including the random suffix AWS appends). You'll need this when writing the IAM policy.

This approach works with both AWS Secrets Manager and AWS Parameter Store.

Step 2 — Create an EKS Cluster

Use eksctl to spin up a cluster. You can pass flags directly or use a YAML config file:

eksctl create cluster \
  --name my-cluster \
  --region us-east-1 \
  --version 1.29 \
  --zones us-east-1a,us-east-1b \
  --managed

eksctl uses CloudFormation under the hood and creates at least two stacks: one for the cluster and one for the managed node group, backed by an EC2 Auto Scaling Group.

Once the cluster is ready, verify connectivity:

kubectl get nodes

Step 3 — Create an IAM OpenID Connect (OIDC) Provider

To allow Kubernetes service accounts to assume IAM roles, you need an OIDC identity provider:

1. Go to the EKS console, open your cluster, and copy the OpenID Connect provider URL.
2. In the IAM console, go to Identity providers and click Add provider.
3. Select OpenID Connect, paste the URL, and click Get thumbprint.
4. Set the audience to sts.amazonaws.com.

Step 4 — Create an IAM Policy and Role

Policy

Create a policy that allows reading the specific secret:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "<your-secret-arn>"
    }
  ]
}

Name it something like APITokenReadAccess.

Role

Create an IAM role:

1. Select Web identity as the trusted entity and choose the OIDC provider you created.
2. Attach the APITokenReadAccess policy.
3. Name the role api-token-access.

Tighten the Trust Relationship

Edit the role's trust policy so only a specific Kubernetes service account can assume it:

{
  "Condition": {
    "StringEquals": {
      "<oidc-provider-url>:sub": "system:serviceaccount:production:nginx"
    }
  }
}

Replace the aud key with sub and set the value to :<service-account-name. This ensures no other service account in the cluster can assume this role.

Step 5 — Create the Kubernetes Namespace and Service Account

Create a folder (e.g., nginx/) to hold your Kubernetes manifests.

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: production

serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/api-token-access

The annotation on the service account is what links it to the IAM role. Without it, the pod won't be able to assume the role and the secret retrieval will fail.

Apply both:

kubectl apply -f nginx/

Step 6 — Install the Secrets Store CSI Driver

The Secrets Store CSI Driver is what mounts secrets from external stores into pods as volumes. There are two ways to install it.

Option A — Plain YAML

Apply the two required Custom Resource Definitions:

kubectl apply -f secret-provider-class-crd.yaml
kubectl apply -f secret-provider-class-pod-status-crd.yaml

Then deploy the driver components: ServiceAccount, ClusterRole, ClusterRoleBinding, DaemonSet, and CSIDriver. The ClusterRole must include secrets access if you want to expose secrets as environment variables.

kubectl apply -f csi-driver/
kubectl logs -l app=secrets-store-csi-driver

Option B — Helm

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

Note: The Helm chart does not enable Kubernetes secret syncing by default. You'll need to add RBAC permissions manually if you want environment variable support.

Step 7 — Install the AWS Secrets and Configuration Provider

This provider bridges the CSI driver with AWS Secrets Manager and Parameter Store. Create and apply the following manifests: ServiceAccount, ClusterRole, ClusterRoleBinding, and a DaemonSet.

kubectl apply -f aws-provider/
kubectl logs -l app=csi-secrets-store-provider-aws

Step 8 — Create a SecretProviderClass

A SecretProviderClass is the custom resource that maps your AWS secret to a Kubernetes-consumable object. Here's what the manifest looks like:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/service-token"
        objectType: "secretsmanager"
        objectAlias: "api-token"
  secretObjects:
    - secretName: api-token
      type: Opaque
      data:
        - objectName: api-token
          key: SECRET_TOKEN

The parameters.objects block tells the provider which secret to fetch from AWS. The secretObjects block syncs it into a native Kubernetes Secret, which is what makes environment variable injection possible.

kubectl apply -f nginx/secret-provider-class.yaml

Step 9 — Deploy Nginx and Consume the Secret

Here's a deployment that mounts the secret as a file and exposes it as an environment variable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: nginx   # must match the annotated service account
      containers:
        - name: nginx
          image: nginx
          volumeMounts:
            - name: secrets-store
              mountPath: /mnt/api-token
              readOnly: true
          env:
            - name: API_TOKEN
              valueFrom:
                secretKeyRef:
                  name: api-token
                  key: SECRET_TOKEN
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: aws-secrets

kubectl apply -f nginx/deployment.yaml

Watch the CSI driver logs after applying. You'll see immediately whether the secret was retrieved successfully or whether there's a permissions error to troubleshoot.

Step 10 — Verify

Check the mounted file:

kubectl exec -it <pod-name> -n production -- cat /mnt/api-token/api-token

Check the environment variable:

kubectl exec -it <pod-name> -n production -- env | grep API_TOKEN

Both should display your secret value from AWS Secrets Manager.

Troubleshooting Common Failures

Pod stuck in ContainerCreating with a CSI volume error. The most common cause is the AWS provider DaemonSet not running on the node where the pod is scheduled. Check with kubectl get pods -n kube-system -l app=csi-secrets-store-provider-aws and confirm there is a running pod on every node.

AccessDeniedException in the provider logs. The IRSA trust policy is the first thing to verify. The StringEquals condition must match :<service-account-name exactly. A mismatch in namespace or service account name silently fails at the STS AssumeRoleWithWebIdentity call.

Secret syncs to the volume but the Kubernetes Secret is not created. The CSI driver must have the syncSecret.enabled=true Helm value set, and the ClusterRole must grant secrets RBAC access. Without this, secretObjects in the SecretProviderClass is silently ignored, so environment variable injection will fail while the mounted file works fine.

Secret value is stale after rotation in Secrets Manager. The CSI driver polls for updates on a rotation interval (default: 2 minutes). Pods that consume the secret via environment variables do not automatically pick up the new value because environment variables are set at container start. Only the mounted file reflects the updated value in-place. For rotation to reach environment variables, the pod must be restarted.

The annotation eks.amazonaws.com/role-arn on the service account is what triggers the token projection for IRSA. If the annotation is missing or points to the wrong ARN, the pod's service account token will not be exchanged for AWS credentials, and every Secrets Manager API call will fail with a 401.

Summary

Conclusion

With this setup, pods consume secrets directly from AWS Secrets Manager without any credentials in manifests, images, or etcd. The IRSA trust relationship scopes access to a single service account, the CSI driver handles volume mounting and Secret sync, and Secrets Manager handles rotation independently of the cluster. Adding new secrets requires updating the SecretProviderClass and redeploying — the IRSA and driver infrastructure does not need to change.