Setup EKS Cluster and Manage Credentials at runtime using CSI driver using SecretProviderClass and Secrets Manager
Assuming you have Configured/Installed AWS CLI, EKSCTL, KUBECTL, HELM.
CSI Basic Information:
- CSI (Container Storage Interface) widely used as a Storage Technology. Created by Google | Mesosphere | Docker.
- It has two two Plugins one runs on the Master Node (Centralized Controller Plugin) and another one on Worker Nodes (Decentralized headless Node Plugin).
- CSI communication protocol is gRPC.
- The communication between Container Orchestration to Controller Plugin (Master) and to Node Plugin (Worker Node) happens using gRPC.
- CSI Drivers: vendor specific compiled into Kubernetes/openshift binaries. To use a CSI driver, a StorageClass needs to be assigned first.
- The CSI driver is then set as the Provisioner for the Storage Class.
- CSI drivers provide three main services, which are: Identity | Controller and Node based.
- Identity: For Plugins related.
- Controller: For Cluster operations related.
- Node: For Container and Local machine(s) operations related.
As shown above, this is how CSI Operates using the Remote Procedure Call by gRPC.
Now we will create the basic EKS Cluster using EKSCTL which in turn runs a Cloud Formation Stack to create the AWS resources.
Here is the basic ClusterConfig yaml file (my-cluster.yaml) to start the EKS Cluster named "basic-cluster".
------------------------------------------------------------------------------------------------------------------
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: basic-cluster
region: ap-southeast-1
version: "1.22"
managedNodeGroups:
- name: mana-ng-pub
instanceType: t3.medium
minSize: 1
desiredCapacity: 1
maxSize: 1
availabilityZones: ["ap-southeast-1a"]
volumeSize: 20
ssh: # Using existing EC2 Key pair
allow: true
publicKeyName: eks-demo-sg # had this key already, can replace this with yours
tags:
nodegroup-role: worker
------------------------------------------------------------------------------------------------------------------
Below is the "SecretProviderClass" yaml (demo-spc.yaml) file.
Note: Can skip this file shown below to give a basic idea about it.
------------------------------------------------------------------------------------------------------------------
---
# apiVersion: secrets-store.csi.x-k8s.io/v1alpha1 # deprecated
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secret-application
spec:
provider: aws # for aws can use this else for any other Clouds can change this value
secretObjects:
- secretName: MySecret #application-api-key # the k8s secret name
type: Opaque
data:
- objectName: MySecret #secret-api-key # reference the corresponding parameter
key: api_key
parameters:
objects: |
- objectName: "MySecret" #(secret-api-key, # the AWS secret)
objectType: "secretsmanager"
------------------------------------------------------------------------------------------------------------------
1.a, Create the EKS Cluster using the below command and verify once done -
➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl create cluster -f my-cluster.yaml
2, Get EKS Cluster Information -
➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl get cluster
2022-05-11 23:59:36 [ℹ] eksctl version 0.96.0
2022-05-11 23:59:36 [ℹ] using region ap-southeast-1
NAME REGION EKSCTL CREATED
basic-cluster ap-southeast-1 True
3, Get EKS Cluster SVC running -
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 47m
4, "Install the ASCP (AWS Secrets and Configuration Provider)" :
The ASCP is available on GitHub in the secrets-store-csi-provider-aws repository.
The repo also contains example YAML files for creating and mounting a secret.
You first install the Kubernetes Secrets Store CSI Driver, and then you install the ASCP.
a. To install the SecretsStore CSI Driver, run the following commands.
For full installation instructions, see Installation in the Secrets Store CSI Driver Book.
➜ AWS_ACCOUNT_DVO_EMAIL $ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
➜ AWS_ACCOUNT_DVO_EMAIL $ helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
To verify that Secrets Store CSI Driver has started, run:
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
NAME READY STATUS RESTARTS AGE
csi-secrets-store-secrets-store-csi-driver-k24z6 3/3 Running 0 31s
b. To install the ASCP, use the YAML file in GitHub repo deployment directory.
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
serviceaccount/csi-secrets-store-provider-aws created
clusterrole.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-rolebinding created
daemonset.apps/csi-secrets-store-provider-aws created
Note: The above command will create the following and the respective code for them is shown below -
- ServiceAccount
- ClusterRole
- ClusterRoleBinding
- DaemonSet
------------------------------------------------------------------------------------------------------------------
# https://kubernetes.io/docs/reference/access-authn-authz/rbac
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-secrets-store-provider-aws
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: csi-secrets-store-provider-aws-cluster-role
rules:
- apiGroups: [""]
resources: ["serviceaccounts/token"]
verbs: ["create"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["get"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: csi-secrets-store-provider-aws-cluster-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: csi-secrets-store-provider-aws-cluster-role
subjects:
- kind: ServiceAccount
name: csi-secrets-store-provider-aws
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
namespace: kube-system
name: csi-secrets-store-provider-aws
labels:
app: csi-secrets-store-provider-aws
spec:
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: csi-secrets-store-provider-aws
template:
metadata:
labels:
app: csi-secrets-store-provider-aws
spec:
serviceAccountName: csi-secrets-store-provider-aws
hostNetwork: true
containers:
- name: provider-aws-installer
image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-6-gee95299-2022.04.14.21.07
imagePullPolicy: Always
args:
- --provider-volume=/etc/kubernetes/secrets-store-csi-providers
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 50m
memory: 100Mi
volumeMounts:
- mountPath: "/etc/kubernetes/secrets-store-csi-providers"
name: providervol
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: HostToContainer
volumes:
- name: providervol
hostPath:
path: "/etc/kubernetes/secrets-store-csi-providers"
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: DirectoryOrCreate
nodeSelector:
kubernetes.io/os: linux
------------------------------------------------------------------------------------------------------------------
3, To create and mount a secret :
a, Set the AWS Region & name of your cluster as shell variables so you can use the min bash commands. For <REGION>, enter the AWS Region where your Amazon EKS cluster runs. For <CLUSTERNAME>, enter the name of your cluster.
➜ AWS_ACCOUNT_DVO_EMAIL $ export REGION=ap-southeast-1 && export CLUSTERNAME="basic-cluster"
b, Create a test secret:
➜ AWS_ACCOUNT_DVO_EMAIL $ aws --region "$REGION" secretsmanager create-secret --name MySecret --secret-string '{"username":"foo", "password":"bar"}'
{
"ARN": "arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8",
"Name": "MySecret",
"VersionId": "8e47ef05-4f60-499a-b842-90903c7082fd"
}
c, Create a Resource Policy for the Pod that Limits its Access to the Secret you created in the previous step.
For <SECRETARN>, use the ARN of the secret. Save the policy ARN in a shell variable.
➜ AWS_ACCOUNT_DVO_EMAIL $ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"], "Resource": ["arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8"]}] }')
d, Create IAM OIDC Provider for Cluster if you don't already have one -
➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once
2022-05-12 00:19:43 [ℹ] eksctl version 0.96.0
2022-05-12 00:19:43 [ℹ] using region ap-southeast-1
2022-05-12 00:19:44 [ℹ] will create IAM Open ID Connect provider for cluster "basic-cluster" in "ap-southeast-1"
2022-05-12 00:19:45 [✔] created IAM Open ID Connect provider for cluster "basic-cluster" in "ap-southeast-1"
e, Create Service Account, Pod uses & associate the Resource Policy created in step3 with that service account.
For this tutorial, for the service account name, you use nginx-deployment-sa -
➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl create iamserviceaccount --name nginx-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts
f, Create SecretProviderClass to specify which secret to mount in Pod.
The following command uses ExampleSecretProviderClass.yaml in the ASCP GitHub repo examples directory to mount the secret you created in step 1 -
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass.yaml
Warning: secrets-store.csi.x-k8s.io/v1alpha1 is deprecated.
Use "secrets-store.csi.x-k8s.io/v1" instead.
secretproviderclass.secrets-store.csi.x-k8s.io/nginx-deployment-aws-secrets created
------------------------------------------------------------------------------------------------------------------
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: nginx-deployment-aws-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "MySecret" # can name as per your secret in AWS Secrets Manager
objectType: "secretsmanager" # can change to AWS Secrets Parameter if using that.
------------------------------------------------------------------------------------------------------------------
g, Deploy your pod. The following command uses ExampleDeployment.yaml in the ASCP GitHub repo examples directory to mount the secret in /mnt/secrets-store in the pod -
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment.yaml
service/nginx-deployment created
deployment.apps/nginx-deployment created
h, To verify Secret has been mounted properly, use the following command and confirm that your secret value appears -
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it $(kubectl get pods | awk '/nginx-deployment/{print $1}' | head -1) cat /mnt/secrets-store/MySecret; echo
OR
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec nginx-deployment-76f76f6b68-5wgjn -- cat /mnt/secrets-store/MySecret; echo
{"username":"foo", "password":"bar"}
i, To Get inside the POD to check the secrets -
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it nginx-deployment-76f76f6b68-5wgjn -- sh
and then run inside the POD -
# cat /mnt/secrets-store/MySecret
{"username":"foo", "password":"bar"}
4, Once all test and verified can delete the EKS Cluster and other things -
➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl delete cluster --name $CLUSTERNAME
---------
To get a shell access to the container running inside the application pod, all you have to do is:
➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it --namespace <your namespace> <your pod name> -- sh
---------
Benefits of using CSI Driver:
1. Can store the secrets directly into the PODS using the CSI ➜ SecretProviderClass ➜ AWS Secrets Manager.
2. No need to have configurations for the application to pass the values.
3. Can retrieve the credentials, api-keys etc at run time from the mounted volumes in Deployment/Pods using the environment(s).
4. Can pass multiple secrets instead of single in the SecretProviderClass and can use them in application using the environment data.
5, And many more..
Ref: