Skip to main content

EKS Cluster and Create CSI Driver to store credentials in AWS Secrets Manager via SecretProviderClass

EKS Cluster and Create CSI Driver to Store Credentials in AWS Secrets Manager

Setup EKS Cluster and Manage Credentials at Runtime using CSI Driver with SecretProviderClass and AWS Secrets Manager

Assuming you have configured/installed AWS CLI, EKSCTL, KUBECTL, and HELM.

📦 CSI Basic Information

  1. CSI (Container Storage Interface) is widely used as a Storage Technology. Created by Google | Mesosphere | Docker.
  2. It has two plugins: one runs on the Master Node (Centralized Controller Plugin) and another on Worker Nodes (Decentralized headless Node Plugin).
  3. CSI communication protocol is gRPC.
  4. The communication between Container Orchestration to Controller Plugin (Master) and to Node Plugin (Worker Node) happens using gRPC.
  5. CSI Drivers: vendor-specific, compiled into Kubernetes/OpenShift binaries. To use a CSI driver, a StorageClass needs to be assigned first.
  6. The CSI driver is then set as the Provisioner for the Storage Class.
  7. CSI drivers provide three main services:
    • Identity - For Plugins related
    • Controller - For Cluster operations related
    • Node - For Container and Local machine(s) operations related
CSI Architecture - How CSI operates using gRPC Remote Procedure Call

CSI Architecture: How CSI operates using the Remote Procedure Call via gRPC

Now we will create a basic EKS Cluster using EKSCTL, which in turn runs a CloudFormation Stack to create the AWS resources.

⚙️ Step 1: EKS Cluster Configuration

Here is the basic ClusterConfig YAML file (my-cluster.yaml) to start the EKS Cluster named "basic-cluster":

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: basic-cluster
  region: ap-southeast-1
  version: "1.22"

managedNodeGroups:
  - name: mana-ng-pub
    instanceType: t3.medium
    minSize: 1
    desiredCapacity: 1
    maxSize: 1
    availabilityZones: ["ap-southeast-1a"]
    volumeSize: 20
    ssh: # Using existing EC2 Key pair
      allow: true
      publicKeyName: eks-demo-sg # Replace with your key
    tags:
      nodegroup-role: worker

🔐 SecretProviderClass Configuration

Below is the "SecretProviderClass" YAML file (demo-spc.yaml).

Note: You can skip this file for now; it's shown below to give you a basic idea about it.

---
# apiVersion: secrets-store.csi.x-k8s.io/v1alpha1 # deprecated
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secret-application
spec:
  provider: aws # For AWS; change for other clouds
  secretObjects:
    - secretName: MySecret # The k8s secret name
      type: Opaque
      data:
        - objectName: MySecret # Reference the corresponding parameter
          key: api_key
  parameters:
    objects: |
      - objectName: "MySecret" # The AWS secret
        objectType: "secretsmanager"

🚀 Step-by-Step Implementation

Step 1a: Create the EKS Cluster

➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl create cluster -f my-cluster.yaml

Verify once the cluster is created successfully.

Step 2: Get EKS Cluster Information

➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl get cluster

2022-05-11 23:59:36 [ℹ] eksctl version 0.96.0
2022-05-11 23:59:36 [ℹ] using region ap-southeast-1

NAME              REGION          EKSCTL CREATED
basic-cluster     ap-southeast-1  True

Step 3: Get EKS Cluster Services Running

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl get svc

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   47m

🔧 Step 4: Install ASCP (AWS Secrets and Configuration Provider)

The ASCP is available on GitHub in the secrets-store-csi-driver-provider-aws repository.

The repository also contains example YAML files for creating and mounting a secret.

You first install the Kubernetes Secrets Store CSI Driver, and then you install the ASCP.

Step 4a: Install the Secrets Store CSI Driver

For full installation instructions, see Installation in the Secrets Store CSI Driver Book.

➜ AWS_ACCOUNT_DVO_EMAIL $ helm repo add secrets-store-csi-driver \
    https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

➜ AWS_ACCOUNT_DVO_EMAIL $ helm install -n kube-system csi-secrets-store \
    secrets-store-csi-driver/secrets-store-csi-driver

To verify that Secrets Store CSI Driver has started, run:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"

NAME                                               READY   STATUS    RESTARTS   AGE
csi-secrets-store-secrets-store-csi-driver-k24z6   3/3     Running   0          31s

Step 4b: Install the ASCP

Use the YAML file in the GitHub repo deployment directory:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f \
    https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

serviceaccount/csi-secrets-store-provider-aws created
clusterrole.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-rolebinding created
daemonset.apps/csi-secrets-store-provider-aws created

Note:

The above command creates the following resources:

  • ServiceAccount
  • ClusterRole
  • ClusterRoleBinding
  • DaemonSet
📄 Click to view complete YAML configuration 
# https://kubernetes.io/docs/reference/access-authn-authz/rbac
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-aws
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-aws-cluster-role
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-aws-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
subjects:
  - kind: ServiceAccount
    name: csi-secrets-store-provider-aws
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: csi-secrets-store-provider-aws
  labels:
    app: csi-secrets-store-provider-aws
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-aws
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-aws
    spec:
      serviceAccountName: csi-secrets-store-provider-aws
      hostNetwork: true
      containers:
        - name: provider-aws-installer
          image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-6-gee95299-2022.04.14.21.07
          imagePullPolicy: Always
          args:
            - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: HostToContainer
      volumes:
        - name: providervol
          hostPath:
            path: "/etc/kubernetes/secrets-store-csi-providers"
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
      nodeSelector:
        kubernetes.io/os: linux

🔐 Step 5: Create and Mount a Secret

Step 5a: Set Environment Variables

Set the AWS Region and name of your cluster as shell variables:

➜ AWS_ACCOUNT_DVO_EMAIL $ export REGION=ap-southeast-1
➜ AWS_ACCOUNT_DVO_EMAIL $ export CLUSTERNAME="basic-cluster"

Step 5b: Create a Test Secret

➜ AWS_ACCOUNT_DVO_EMAIL $ aws --region "$REGION" secretsmanager \
    create-secret --name MySecret \
    --secret-string '{"username":"foo", "password":"bar"}'

{
    "ARN": "arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8",
    "Name": "MySecret",
    "VersionId": "8e47ef05-4f60-499a-b842-90903c7082fd"
}

Step 5c: Create IAM Resource Policy

Create a Resource Policy for the Pod that limits its access to the secret you created in the previous step:

➜ AWS_ACCOUNT_DVO_EMAIL $ POLICY_ARN=$(aws --region "$REGION" \
    --query Policy.Arn --output text iam create-policy \
    --policy-name nginx-deployment-policy \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Action": [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ],
        "Resource": [
          "arn:aws:secretsmanager:ap-southeast-1:1233444455566:secret:MySecret-rbkra8"
        ]
      }]
    }')

Step 5d: Create IAM OIDC Provider

Create an IAM OIDC Provider for the cluster if you don't already have one:

➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl utils associate-iam-oidc-provider \
    --region="$REGION" --cluster="$CLUSTERNAME" --approve

2022-05-12 00:19:43 [ℹ] eksctl version 0.96.0
2022-05-12 00:19:43 [ℹ] using region ap-southeast-1
2022-05-12 00:19:44 [ℹ] will create IAM Open ID Connect provider for cluster "basic-cluster"
2022-05-12 00:19:45 [✔] created IAM Open ID Connect provider for cluster "basic-cluster"

Note: Only run this once per cluster

Step 5e: Create Service Account

Create a Service Account that the Pod uses and associate the Resource Policy created in step 5c with that service account:

➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl create iamserviceaccount \
    --name nginx-deployment-sa \
    --region="$REGION" \
    --cluster "$CLUSTERNAME" \
    --attach-policy-arn "$POLICY_ARN" \
    --approve \
    --override-existing-serviceaccounts

Step 5f: Create SecretProviderClass

Create a SecretProviderClass to specify which secret to mount in the Pod:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f \
    https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass.yaml

Warning: secrets-store.csi.x-k8s.io/v1alpha1 is deprecated.
Use "secrets-store.csi.x-k8s.io/v1" instead.
secretproviderclass.secrets-store.csi.x-k8s.io/nginx-deployment-aws-secrets created

SecretProviderClass YAML:

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "MySecret"    # Name as per your secret in AWS Secrets Manager
        objectType: "secretsmanager" # Change to parameter store if using that

Step 5g: Deploy Your Pod

Deploy your pod. The following command mounts the secret in /mnt/secrets-store in the pod:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl apply -f \
    https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment.yaml

service/nginx-deployment created
deployment.apps/nginx-deployment created

Step 5h: Verify Secret Mounting

To verify the secret has been mounted properly, use the following command and confirm that your secret value appears:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it \
    $(kubectl get pods | awk '/nginx-deployment/{print $1}' | head -1) \
    cat /mnt/secrets-store/MySecret; echo

OR

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec nginx-deployment-76f76f6b68-5wgjn -- \
    cat /mnt/secrets-store/MySecret; echo

{"username":"foo", "password":"bar"}

Step 5i: Access Pod Shell

To get inside the POD to check the secrets:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it nginx-deployment-76f76f6b68-5wgjn -- sh

# cat /mnt/secrets-store/MySecret
{"username":"foo", "password":"bar"}

🧹 Step 6: Cleanup

Once all tests are verified, you can delete the EKS Cluster and other resources:

➜ AWS_ACCOUNT_DVO_EMAIL $ eksctl delete cluster --name $CLUSTERNAME

💡 Additional Tips

🔹 Get Shell Access to Container

To get a shell access to the container running inside the application pod:

➜ AWS_ACCOUNT_DVO_EMAIL $ kubectl exec -it \
    --namespace <your-namespace> <your-pod-name> -- sh

Benefits of Using CSI Driver:

  1. Can store secrets directly into the PODS using CSI → SecretProviderClass → AWS Secrets Manager.
  2. No need for application configuration to pass the values.
  3. Retrieve credentials at runtime from mounted volumes in Deployment/Pods using environment variables.
  4. Pass multiple secrets instead of a single one in the SecretProviderClass and use them in the application using environment data.
  5. And many more...

📚 References:

Labels:
Location: 12 Siglap Link, Singapore 448870

Comments

Post a Comment

Popular posts from this blog

Hacking via Cloning Site Using Kali Linux

Hacking via Cloning Site Using Kali Linux Hacking via Cloning Site Using Kali Linux  SET Attack Method : SET stands for Social Engineering Toolkist , primarily written by  David Kennedy . The Social-Engineer Toolkit (SET) is specifically designed to perform advanced attacks against the human element. SET was designed to be released with the  http://www.social-engineer.org  launch and has quickly became a standard tool in a penetration testers arsenal. The attacks built into the toolkit are designed to be targeted and focused attacks against a person or organization used during a penetration test. Actually this hacking method will works perfectly with DNS spoofing or Man in the Middle Attack method. Here in this tutorial I’m only writing how-to and step-by-step to perform the basic attack , but for the rest you can modified it with your own imagination. In this tutorial we will see how this attack methods can owned your com...
Post a Comment
Read more

Defacing Sites via HTML Injections (XSS)

Defacing Sites via HTML Injections Defacing Sites via HTML Injections What Is HTML Injection: "HTML Injection" is called as the Virtual Defacement Technique and also known as the "XSS" Cross Site Scripting. It is a very common vulnerability found when searched for most of the domains. This kind of a Vulnerability allows an "Attacker" to Inject some code into the applications affected in order to bypass access to the "Website" or to Infect any particular Page in that "Website". HTML injections = Cross Site Scripting, It is a Security Vulnerability in most of the sites, that allows an Attacker to Inject HTML Code into the Web Pages that are viewed by other users. XSS Attacks are essentially code injection attacks into the various interpreters in the browser. These attacks can be carried out using HTML, JavaScript, VBScript, ActiveX, Flash and other clinet side Languages. Well crafted Malicious Code can even hep the ...
Post a Comment
Read more

Hacking DNN Based Web Sites

Hacking DNN Based Web Sites Hacking DNN Based Web Sites Hacking DNN (Dot Net Nuke) CMS based websites is based on the Security Loop Hole in the CMS. For using that exploit we will see the few mentioned points which illustrates us on how to hack any live site based on Dot Net Nuke CMS. Vulnerability : This is the know Vulnerability in Dot Net Nuke (DNN) CMS. This allows aone user to Upload a File/Shell Remotely to hack that Site which is running on Dot Net Nuke CMS. The Link's for more Information regarding this Vulnerability is mentioned below -                                  http://www.exploit-db.com/exploits/12700/ Getting Started : Here we will use the Google Dork to trace the sites that are using DNN (Dot Net Nuke) CMS and are vulnerable to Remote File Upload. How To Do It : Here, I an mentioning the few points on how to Search for the existing Vulnerability in DNN. Let'...
Post a Comment
Read more

Excellent tricks and techniques of Google Hacks

Frontpage.. very nice clean search results listing !! I magine with me that you can steal or know the password of any web site designed by "Frontpage". But the file containing the password might be encrypted; to decrypt the file download the program " john the ripper". To see results; just write in the ( http://www.google.com/ ) search engine the code: "# -FrontPage-" inurl:service.pwd ============================================== This searches the password for "Website Access Analyzer", a Japanese software that creates webstatistics. To see results; just write in the ( http://www.google.com/ ) search engine the code: "AutoCreate=TRUE password=*" ============================================== This is a query to get inline passwords from search engines (not just Google), you must type in the query followed with the the domain name without the .com or .net. To see results; just write in the ( http://www.google.co...
Post a Comment
Read more

Hacking via BackTrack using SET Attack Method

Hacking via BackTrack using SET Attack Method Hacking via BackTrack using SET Attack  1. Click on Applications, BackTrack, Exploit Tools, Social Engineering Tools, Social Engineering Toolkit then select set.
Post a Comment
Read more