Kubernetes Secrets Encryption

1Configure Luna HSM

2Set up Kubernetes Cluster

1Ensure that the HSM is set up, initialized, provisioned, and ready for deployment. For more information, refer to Luna HSM documentation.

2Create a partition that will be later on used by Kubernetes.

3Create and exchange certificate between the Luna Network HSM and client system. Register client and assign partition to create an NTLS connection.

4Initialize Crypto Officer and Crypto User roles for the registered partition.

5Run the following command to verify that the partition has been successfully registered and configured:

/usr/safenet/lunaclient/bin/lunacm 

Upon successful execution, you should observe an output similar to the example provided below:

lunacm (64-bit) v10.4.0-417. Copyright (c) 2021 Thales Group. All rights reserved. 
Available HSMs:

Slot Id ->                           0
Label ->                             KubeKMS01
Serial Number ->                     1238696044948
Model ->                             LunaSA 7.4.0
Firmware Version ->                  7.4.0
Configuration ->                     Luna User Partition With SO (PW) Key Export With Cloning Mode
Slot Description ->                  Net Token Slot
FM HW Status ->                      Non-FM

Slot Id ->                           1
Label ->                             KubeKMS02
Serial Number ->                     1238696044908
Model ->                             LunaSA 7.4.0
Firmware Version ->                  7.4.0
Configuration ->                     Luna User Partition With SO (PW) Signing With Cloning Mode
Slot Description ->                  Net Token Slot
FM HW Status ->                      Non-FM

Slot Id ->                           8
HSM Label ->                         KubeKMS-HA
HSM Serial Number ->                 11238696044948
HSM Model ->                         LunaVirtual
HSM Firmware Version ->              7.4.0
HSM Configuration ->                 Luna Virtual HSM (PW) Signing With Cloning Mode
HSM Status ->                        N/A - HA Group
HSM Certificates ->                  *** Test Certs *** 

Current Slot Id: 0 

Note

Refer to Luna HSM documentation for detailed steps on creating NTLS connection, initializing the partitions, and assigning various user roles.

Note

For proper configuration of a PED-based Luna HSM, it is recommended to activate partition policies 22 and 23, allowing for both activation and auto-activation.

1Configure Luna HSM with K8S-KMS-Plugin

2Deploy K8S-KMS-Plugin as KMS Provider

3Verify Kubernetes Secrets encryption using KMS Provider

4Switch from a local encryption provider to the KMS provider

1Access the master host using a root account or an account with administrative privileges.

2Define the KMSPATH and create the necessary directory structure:

export KMSPATH=/opt/kms
mkdir -p $KMSPATH/src/github.com/thalescpl-io
cd $KMSPATH/src/github.com/thalescpl-io

3Execute the following commands to clone the k8s-kms-plugin repository and check out the specified branch:

git clone https://github.com/thalescpl-io/k8s-kms-plugin.git
cd k8s-kms-plugin
git checkout -b luna_fips_mode_fix origin/luna_fips_mode_fix
cd $KMSPATH/src/github.com/thalescpl-io

4Copy the Luna Minimal Client package to the k8s-kms-plugin directory:

cp /home/LunaClient-Minimal-10.2.0-111.x86_64.tar k8s-kms-plugin/

5Copy the Luna Configuration file (/etc/Chrystoki.conf) to the k8s-kms-plugin directory:

cp /etc/Chrystoki.conf k8s-kms-plugin/

6Modify the Chrystoki.conf file located in the k8s-kms-plugin directory to update the library paths and the Chrystoki2 and Secure Trusted Channel sections as follows:

Chrystoki2 = {
    LibUNIX = /usr/safenet/lunaclient/libs/64/libCryptoki2.so;
    LibUNIX64 = /usr/safenet/lunaclient/libs/64/libCryptoki2_64.so;
}

...

Secure Trusted Channel = {
    ClientTokenLib = /usr/safenet/lunaclient/libs/64/libSoftToken.so;
    SoftTokenDir = /usr/safenet/lunaclient/stc/token;
    ClientIdentitiesDir = /usr/safenet/lunaclient/stc/client_identities;
    PartitionIdentitiesDir = /usr/safenet/lunaclient/stc/partition_identities;
}

7Copy the necessary certificates and keys to the k8s-kms-plugin directory to enable connectivity to Luna HSM:

cp -r /usr/safenet/lunaclient/cert/server k8s-kms-plugin/
cp -r /usr/safenet/lunaclient/cert/client k8s-kms-plugin/

8Create a start.sh file in the k8s-kms-plugin directory with the necessary flags for Luna HSM:

• Navigate to the k8s-kms-plugin directory:

cd k8s-kms-plugin/

• Create the start.sh file:

echo '#!/usr/bin/env bash
/k8s-kms-plugin serve --provider luna --p11-lib /usr/safenet/lunaclient/libs/64/libCryptoki2_64.so --p11-label KubeKMS-HA --p11-pin userpin1' > start.sh

• Ensure the script has execute permissions:

chmod +x start.sh

Note

Ensure the p11-label and p11-pin are replaced with the appropriate partition name and password, respectively.

9Specify additional options for k8s-kms-plugin: The k8s-kms-plugin provides several additional options which can be specified as per your requirements:

10Create a new Dockerfile or edit the existing one in the k8s-kms-plugin directory. This Dockerfile will be used to create a Docker image containing k8s-kms-plugin and all required resources for Luna HSM Client to communicate with the Luna HSM partition. Ensure that you provide the correct file name and path for all resources being copied from the host to the Docker image.

# Dockerfile

## Build Stage
FROM goboring/golang:1.14.6b4 as build
WORKDIR /app
ADD go.mod /app/go.mod
ADD go.sum /app/go.sum  
ADD tools.go /app/pkg/tools.go
ADD vendor /app/vendor
ADD pkg /app/pkg
ADD apis /app/apis
ADD cmd/ /app/cmd/

ENV GOOS linux
ENV GOARCH amd64
ENV CGO_ENABLED 1
ENV GOFLAGS -mod=vendor
RUN go build -o k8s-kms-plugin ./cmd/k8s-kms-plugin

## Plugin Server
FROM goboring/golang:1.14.6b4 as kms-server
WORKDIR /
COPY --from=build /app/k8s-kms-plugin /k8s-kms-plugin
COPY LunaClient-Minimal-10.2.0-111.x86_64.tar /tmp/
RUN mkdir -p /usr/safenet/lunaclient
RUN mkdir -p /usr/safenet/lunaclient/cert/client
RUN mkdir -p /usr/safenet/lunaclient/cert/server
RUN tar -xvf /tmp/LunaClient-Minimal-10.2.0-111.x86_64.tar --strip 1 -C /usr/safenet/lunaclient
RUN cp /usr/safenet/lunaclient/openssl.cnf /usr/safenet/lunaclient/bin

ENV ChrystokiConfigurationPath=/etc
COPY Chrystoki.conf /etc/Chrystoki.conf 
COPY server/CAFile.pem /usr/safenet/lunaclient/cert/server
COPY client/k8s-master.kube.comKey.pem /usr/safenet/lunaclient/cert/client
COPY client/k8s-master.kube.com.pem /usr/safenet/lunaclient/cert/client
COPY start.sh /start.sh
RUN chmod +x /start.sh

ENTRYPOINT ["/start.sh"]

11Create an image using the Dockerfile with the following command:

docker build . -t kms-server

You will see a confirmation message when the image is built successfully.

12List all Docker images to ensure that the kms-server image has been created successfully. Use the following command:

docker images

The output will display a list of images, including kms-server.

13Create a pod manifest YAML file named kms-plugin.yaml. This file will define the Kubernetes pod configuration for the k8s-kms-plugin server. Ensure that the file contains the following contents:

# kms-plugin.yaml

apiVersion: v1
kind: Pod
metadata:
  name: k8s-kms-plugin-server
  labels:
    app.kubernetes.io/name: k8s-kms-plugin-server
spec:
  containers:
    - name: plugin-server
      image: kms-server
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: /certs/
          name: certstore
        - mountPath: /tmp/run
          name: socket
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
    - name: certstore
      emptyDir: {}
    - name: ca
      hostPath:
        path: /etc/
    - name: socket
      hostPath:
        path: /tmp/run
        type: DirectoryOrCreate
status: {}

This configuration ensures that the k8s-kms-plugin server will run as a pod in your Kubernetes cluster with the necessary volume mounts for certificates and socket communication.

14Copy the kms-plugin.yaml file to the Kubernetes pod manifest location used by the kubelet service for running all static pods. Use the following command:

cp kms-plugin.yaml /etc/kubernetes/manifests/

Note

Ensure that the path /etc/kubernetes/manifests/ matches the static pod manifest location for your cluster deployment. This location may vary, so confirm you are using the correct directory.

15Kubernetes will automatically deploy the k8s-kms-plugin pod from the manifest location. Verify the deployment status with the following command:

kubectl get pods

16On the master host, verify that the socket file is created under the /tmp/run directory specified in kms-plugin.yaml. Use the following command:

ls -ltr /tmp/run

17Connect to the pod to verify that k8s-kms-plugin is integrated with Luna HSM and that the encryption key is generated. Execute the following command on the master host:

kubectl exec -it k8s-kms-plugin-server-k8s-master.kube.com -- /bin/bash

18Ensure that the pod has access to the Luna HSM partition by running the following command inside the pod:

/usr/safenet/lunaclient/bin/64/vtl listslots

19Verify that the encryption key has been generated on the Luna HSM via the KMS-Plugin by checking the contents of the registered Luna HSM partition. Run the following command inside the pod:

/usr/safenet/lunaclient/bin/64/cmu list

By completing these steps, you ensure that the KMS plugin is configured to use the Luna HSM. The UNIX socket hsm-plugin-server.sock will be created on the master node where the API Server is running and will be ready to serve requests from the API Server.

1When the KMS-Plugin is ready to be used as an encryption provider, create the encryption configuration file encryption-config.yaml on the master node where the API server is running. The file should contain the following content:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
      - kms:
          name: k8s-kms-plugin
          endpoint: unix:///tmp/run/hsm-plugin-server.sock
          cachesize: 100
          timeout: 3s
      - identity: {}

2Save the encryption-config.yaml file at a suitable location on the Master host. For example, you can save it at:

/etc/kubernetes/kms/encryption-config.yaml

3Ensure that the file is correctly placed and accessible by the API server for the next steps in configuring Kubernetes Secrets encryption.

4To configure the API server to use the encryption configuration file created in the previous step, follow these instructions:

• Open the kube-apiserver.yaml file from the static pod manifest location (e.g., /etc/kubernetes/manifests/kube-apiserver.yaml).

• Edit the file to add the --encryption-provider-config flag to the command list of kube-apiserver. The flag must point to the encryption-config.yaml file. Ensure the correct indentation is maintained.

    spec:
      containers:
      - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/kms/encryption-config.yaml
        - --encryption-provider-config-automatic-reload=true
        - --advertise-address=10.164.76.80
        - --allow-privileged=true
        - --authorization-mode=Node,RBAC
        ...

• For Kubernetes version 1.29 and later, the v1 implementation of KMS is disabled by default. To enable this feature, add the --feature-gates=KMSv1=true flag to kube-apiserver.yaml. Ensure to set both flags when using Kubernetes v1.29 and later.

    spec:
      containers:
      - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/kms/encryption-config.yaml
        - --feature-gates=KMSv1=true
        - --encryption-provider-config-automatic-reload=true
        - --advertise-address=10.164.76.80
        - --allow-privileged=true
        - --authorization-mode=Node,RBAC
        ...

5To allow the kube-apiserver pod to communicate with the k8s-kms-plugin, access to the directories where the encryption-config.yaml file and UNIX socket are located is required. Add mount points in kube-apiserver.yaml as shown below, ensuring correct indentation:

• Add Volume Mounts:

    ...
    volumeMounts:
    - mountPath: /etc/kubernetes/kms
      name: kms
      readOnly: true
    - mountPath: /tmp/run
      name: socket
    ...

• Add Volumes:

    ...
    volumes:
    - hostPath:
        path: /etc/kubernetes/kms
        type: DirectoryOrCreate
      name: kms
    - hostPath:
        path: /tmp/run
        type: DirectoryOrCreate
      name: socket
    ...

6After making the necessary changes to the kube-apiserver.yaml file, save and close the file. The API server will automatically restart upon saving the changes. Ensure that your cluster comes back online without any failures and that the API server is READY and running.

kubectl get pods --all-namespaces

This command will help you verify that all pods, including the API server, are running as expected.

7If the API server is configured successfully to use the k8s-kms-plugin, the connection will be established. You can verify this by running the following command on the Master node to check the UNIX socket:

ss -a --unix -p | grep hsm-plugin-server.sock

This command will list the active UNIX sockets, and you should see the hsm-plugin-server.sock if the connection is established correctly.

1Create a new Secret called mysecret in the default namespace. The encrypted Secret gets saved in etcd.

kubectl create secret generic mysecret -n default --from-literal=mykey=mys3cr3t

2Use the etcdctl command to retrieve Secrets from etcd:

• Set up an alias for etcdctl3:

alias etcdctl3="ETCDCTL_API=3 /path/to/etcdctl --endpoints=https://127.0.0.1:2379 --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key --cacert=/etc/kubernetes/pki/etcd/ca.crt"

Note

Ensure to replace /path/to/etcdctl with the actual path to your etcdctl binary.

• Retrieve the Secret from etcd:

etcdctl3 get /registry/secrets/default/mysecret

Note

This command will fetch the Secret from etcd, and you should see the encrypted data, indicating that the KMS provider is correctly encrypting Secrets.

3Ensure that the stored Secret is prefixed with k8s:enc:kms:v1:k8s-kms-plugin, which indicates that the KMS Provider has encrypted the resulting data.

etcdctl3 get /registry/secrets/default/mysecret | hexdump -C

4Ensure that the Secret is correctly decrypting when retrieved via the API Server.

kubectl describe secret mysecret -n default

5Run the following command to encrypt all pre-existing Secrets. The command reads all Secrets and then encrypts all Secrets using the KMS Provider.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Note

If an error occurs due to a conflicting write, retry the command. For larger clusters, it is recommended to subdivide the Secrets by namespace or script an update.

1On the Master host, edit the encryption-config.yaml file. Add the KMS provider as the first entry in the configuration file, as shown in the example below:

     apiVersion: apiserver.config.k8s.io/v1
     kind: EncryptionConfiguration
     resources:
       - resources:
         - secrets
         providers:
         - kms:
             name: k8s-kms-plugin
             endpoint: unix:///tmp/run/socket.sock
             cachesize: 100
             timeout: 3s
         - aescbc:
             keys:
               - name: key1
                 secret: <BASE 64 ENCODED SECRET>

Note

In this configuration, key1 is the name of your key and <BASE 64 ENCODED SECRET> is the actual key of the native encryption provider.

Note

Ensure that the Unix Socket path is mounted in the Kube-API server manifest YAML.

2After updating the encryption configuration file, restart all kube-apiserver processes to apply the changes.

Note

Command to restart kube-apiserver processes will vary based on your setup.

Note

Ensure you follow the proper procedure to restart the kube-apiserver on your Kubernetes setup.

3Run the following command to force all Secrets to be re-encrypted using the KMS Provider:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -