EKS Cluster and Create CSI Driver to store credentials in AWS Secrets Manager via SecretProviderClass

EKS Cluster | CSI Driver | SecretProviderClass | AWS Secrets Manager

Setup EKS Cluster and Manage Credentials at runtime using CSI driver using SecretProviderClass and Secrets Manager

Assuming you have Configured/Installed AWS CLI, EKSCTL, KUBECTL, HELM.

CSI Basic Information:

  1. CSI (Container Storage Interface) widely used as a Storage Technology. Created by Google | Mesosphere | Docker. 
  2. It has two two Plugins one runs on the Master Node (Centralized Controller Plugin) and another one on Worker Nodes (Decentralized headless Node Plugin). 
  3. CSI communication protocol is gRPC. 
  4. The communication between Container Orchestration to Controller Plugin (Master) and to Node Plugin (Worker Node) happens using gRPC
  5. CSI Drivers: vendor specific compiled into Kubernetes/openshift binaries. To use a CSI driver, a StorageClass needs to be assigned first. 
  6. The CSI driver is then set as the Provisioner for the Storage Class.
  7. CSI drivers provide three main services, which are: Identity | Controller and Node based.
  • Identity: For Plugins related.
  • Controller: For Cluster operations related.
  • Node: For Container and Local machine(s) operations related.

As shown above, this is how CSI Operates using the Remote Procedure Call by gRPC.

Now we will create the basic EKS Cluster using EKSCTL which in turn runs a Cloud Formation Stack to create the AWS resources.

Here is the basic ClusterConfig yaml file (my-cluster.yaml) to start the EKS Cluster named "basic-cluster".

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
  name: basic-cluster
  region: ap-southeast-1
  version: "1.22"
  - name: mana-ng-pub
    instanceType: t3.medium
    minSize: 1
    desiredCapacity: 1
    maxSize: 1
    availabilityZones: ["ap-southeast-1a"]
    volumeSize: 20
    ssh: # Using existing EC2 Key pair
      allow: true
      publicKeyName: eks-demo-sg # had this key already, can replace this with yours
      nodegroup-role: worker

Below is the "SecretProviderClass" yaml (demo-spc.yaml) file.
Note: Can skip this file shown below to give a basic idea about it. 

# apiVersion: secrets-store.csi.x-k8s.io/v1alpha1 # deprecated
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
  name: aws-secret-application
  provider: aws # for aws can use this else for any other Clouds can change this value
    - secretName: MySecret #application-api-key  # the k8s secret name
      type: Opaque
        - objectName: MySecret #secret-api-key  # reference the corresponding parameter
          key: api_key
    objects: |
      - objectName: "MySecret" #(secret-api-key, # the AWS secret)
        objectType: "secretsmanager"


1.a, Create the EKS Cluster using the below command and verify once done -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl create cluster -f my-cluster.yaml

2, Get EKS Cluster Information -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl get cluster
2022-05-11 23:59:36 [ℹ]  eksctl version 0.96.0
2022-05-11 23:59:36 [ℹ]  using region ap-southeast-1
basic-cluster ap-southeast-1  True

3, Get EKS Cluster SVC running -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl get svc
kubernetes   ClusterIP   <none>        443/TCP   47m

4, "Install the ASCP (AWS Secrets and Configuration Provider)" :
The ASCP is available on GitHub in the secrets-store-csi-provider-aws repository.
The repo also contains example YAML files for creating and mounting a secret.
You first install the Kubernetes Secrets Store CSI Driver, and then you install the ASCP.

a. To install the SecretsStore CSI Driver, run the following commands.
For full installation instructions, see Installation in the Secrets Store CSI Driver Book.
➜  AWS_ACCOUNT_DVO_EMAIL $ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
➜  AWS_ACCOUNT_DVO_EMAIL $ helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

To verify that Secrets Store CSI Driver has started, run:
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
NAME                                               READY   STATUS    RESTARTS   AGE
csi-secrets-store-secrets-store-csi-driver-k24z6   3/3     Running   0          31s

b. To install the ASCP, use the YAML file in GitHub repo deployment directory.
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
serviceaccount/csi-secrets-store-provider-aws created
clusterrole.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-rolebinding created
daemonset.apps/csi-secrets-store-provider-aws created

Note: The above command will create the following and the respective code for them is shown below -
- ServiceAccount
- ClusterRole
- ClusterRoleBinding
- DaemonSet
# https://kubernetes.io/docs/reference/access-authn-authz/rbac
apiVersion: v1
kind: ServiceAccount
  name: csi-secrets-store-provider-aws
  namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
  name: csi-secrets-store-provider-aws-cluster-rolebinding
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
- kind: ServiceAccount
  name: csi-secrets-store-provider-aws
  namespace: kube-system

apiVersion: apps/v1
kind: DaemonSet
  namespace: kube-system
  name: csi-secrets-store-provider-aws
    app: csi-secrets-store-provider-aws
    type: RollingUpdate
      app: csi-secrets-store-provider-aws
        app: csi-secrets-store-provider-aws
      serviceAccountName: csi-secrets-store-provider-aws
      hostNetwork: true
        - name: provider-aws-installer
          image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-6-gee95299-2022.
          imagePullPolicy: Always
              - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
              cpu: 50m
              memory: 100Mi
              cpu: 50m
              memory: 100Mi
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: HostToContainer
        - name: providervol
            path: "/etc/kubernetes/secrets-store-csi-providers"
        - name: mountpoint-dir
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
        kubernetes.io/os: linux

3, To create and mount a secret :
a, Set the AWS Region & name of your cluster as shell variables so you can use the min bash commands. For <REGION>, enter the AWS Region where your Amazon EKS cluster runs. For <CLUSTERNAME>, enter the name of your cluster.
➜  AWS_ACCOUNT_DVO_EMAIL $ export REGION=ap-southeast-1 && export CLUSTERNAME="basic-cluster"

b, Create a test secret:
➜  AWS_ACCOUNT_DVO_EMAIL $ aws --region "$REGION" secretsmanager  create-secret --name MySecret --secret-string '{"username":"foo", "password":"bar"}'
    "ARN": "arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8",
    "Name": "MySecret",
    "VersionId": "8e47ef05-4f60-499a-b842-90903c7082fd"

c, Create a Resource Policy for the Pod that Limits its Access to the Secret you created in the previous step.
For <SECRETARN>, use the ARN of the secret. Save the policy ARN in a shell variable.
➜  AWS_ACCOUNT_DVO_EMAIL $ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"], "Resource": ["arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8"]}] }')

d, Create IAM OIDC Provider for Cluster if you don't already have one -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once
2022-05-12 00:19:43 [ℹ]  eksctl version 0.96.0
2022-05-12 00:19:43 [ℹ]  using region ap-southeast-1
2022-05-12 00:19:44 [ℹ]  will create IAM Open ID Connect provider for cluster "basic-cluster" in "ap-southeast-1"
2022-05-12 00:19:45 [✔]  created IAM Open ID Connect provider for cluster "basic-cluster" in "ap-southeast-1"

e, Create Service Account, Pod uses & associate the Resource Policy created in step3 with that service account.
For this tutorial, for the service account name, you use nginx-deployment-sa -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl create iamserviceaccount --name nginx-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

f, Create SecretProviderClass to specify which secret to mount in Pod.
The following command uses ExampleSecretProviderClass.yaml in the ASCP GitHub repo examples directory to mount the secret you created in step 1 -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass.yaml
Warning: secrets-store.csi.x-k8s.io/v1alpha1 is deprecated. 
Use "secrets-store.csi.x-k8s.io/v1" instead.
secretproviderclass.secrets-store.csi.x-k8s.io/nginx-deployment-aws-secrets created

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
  name: nginx-deployment-aws-secrets
  provider: aws
    objects: |
        - objectName: "MySecret" # can name as per your secret in AWS Secrets Manager
          objectType: "secretsmanager" # can change to AWS Secrets Parameter if using that.

g, Deploy your pod. The following command uses ExampleDeployment.yaml in the ASCP GitHub repo examples directory to mount the secret in /mnt/secrets-store in the pod -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment.yaml
service/nginx-deployment created
deployment.apps/nginx-deployment created

h, To verify Secret has been mounted properly, use the following command and confirm that your secret value appears -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it $(kubectl get pods | awk '/nginx-deployment/{print $1}' | head -1) cat /mnt/secrets-store/MySecret; echo
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec nginx-deployment-76f76f6b68-5wgjn -- cat /mnt/secrets-store/MySecret; echo
{"username":"foo", "password":"bar"}

i, To Get inside the POD to check the secrets -
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it nginx-deployment-76f76f6b68-5wgjn -- sh
and then run inside the POD -
# cat /mnt/secrets-store/MySecret
{"username":"foo", "password":"bar"}

4, Once all test and verified can delete the EKS Cluster and other things -
➜  AWS_ACCOUNT_DVO_EMAIL $ eksctl delete cluster --name $CLUSTERNAME

To get a shell access to the container running inside the application pod, all you have to do is:
➜  AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it --namespace <your namespace> <your pod name> -- sh

Benefits of using CSI Driver:

1. Can store the secrets directly into the PODS using the CSI ➜ SecretProviderClass ➜ AWS Secrets Manager.
2. No need to have configurations for the application to pass the values.
3. Can retrieve the credentials, api-keys etc at run time from the mounted volumes in Deployment/Pods using the environment(s).
4. Can pass multiple secrets instead of single in the SecretProviderClass and can use them in application using the environment data.
5, And many more..