Kubernetes Secret Encryption Using KMS v2

Ensure that the HSM is set up, initialized, provisioned, and ready for deployment. Refer to the Luna HSM Product Documentation for more information.

Create a partition on the Luna HSM for use with Kubernetes.

Register a client for the system and assign the client to the partition to create an NTLS connection if you are using a Luna Network HSM. Initialize the Crypto Officer and Crypto User roles for the registered partition.

Ensure that the partition is successfully registered and configured. The command to see the registered partitions is:

# /usr/safenet/lunaclient/bin/lunacm

Enable partition policies 22 and 23 to allow activation and auto-activation for PED-authenticated HSM.

Execute the following command on your workstation to create the RSA keys:

# /usr/safenet/lunaclient/bin/cmu generatekeypair -modulusBits=2048 -publicExponent=65537 -label=rsa01 -encrypt=1 -decrypt=1 -wrap=1 -unwrap=1 -id=6E85A75DF42ADB4766544ECC78501E75

Where label is your key name (you can provide any label for your key) and id is a 16-byte hex value (GUID).

id or label is the attribute that K8S-KMS-Plugin refers to when retrieving the key from the Luna HSM, so ensure that every KEK has a unique ID or label.

You can use an online GUID generator to generate a unique ID: https://www.guidgenerator.com

You can also use one of the following Linux commands to generate a 16-byte hex value and use it as the ID:
# head -c16 </dev/urandom | xxd -p -u
# xxd -len 16 -plain /dev/urandom

Provide the partition password when prompted and select PKCS for the mechanism type.

View the generated key pair by running the following command:

# /usr/safenet/lunaclient/bin/cmu list

Provide the partition password when prompted.

Download the latest GA release of K8S-KMS-Plugin compatible with your Linux distribution from the plugin releases page.

# wget https://github.com/eclipse-keysealer/k8s-kms-plugin/releases/download/v0.8.0-test2-ga-aarch64-riscv64/k8s-kms-plugin-0.8.0.test2_ga_aarch64_riscv64-1.x86_64.rpm

Install the downloaded RPM package using the package manager appropriate for your Linux distribution.

# dnf install ./k8s-kms-plugin-0.8.0.test2_ga_aarch64_riscv64-1.x86_64.rpm

The binary is installed in your PATH. To confirm this, run the following command; it displays the help menu.

# k8s-kms-plugin --help

Create a directory for keeping all plugin-related configuration in one place.

# mkdir -p /etc/kms-plugin

Create the environment configuration file that the K8S-KMS-Plugin service reads at startup.

# vi /etc/kms-plugin/k8s-kms-plugin.yaml

Add the following contents, and adjust the values to match your environment:

k8s-kms-plugin:
  log-level: "trace"
  serve:
    socket: "/tmp/run/hsm-plugin-server.sock"
    provider: "luna"
    p11-lib: "/usr/safenet/lunaclient/lib/libCryptoki2_64.so"
    p11-label: "TPA-HA"
    p11-pin: "userpin1"
    p11-key-label: "rsa01"
    algorithm: "rsa-oaep"

Configuration parameter reference:

Create a user and provide the required permissions to run the plugin service. Skip this step if your service runs as the root user.

# useradd kmsuser
# gpasswd --add kmsuser hsmusers

Create a directory to store the socket file and set ownership and permissions for kmsuser.

# mkdir -p /tmp/run
# chown -R kmsuser:kmsuser /tmp/run
# chmod -R 777 /tmp/run/

Create a service to run K8S-KMS-Plugin as a systemd service using a non-privileged user.

# vi /etc/systemd/system/k8s-kms-plugin.service

Add the following contents to the k8s-kms-plugin.service file:

[Unit]
Description=k8s-kms-plugin Service
After=network.target

[Service]
Type=simple
User=kmsuser
Group=kmsuser
Restart=always
ExecStart=/usr/bin/k8s-kms-plugin serve --config /etc/kms-plugin/k8s-kms-plugin.yaml

[Install]
WantedBy=multi-user.target

Start the K8S-KMS-Plugin service.

# systemctl daemon-reload
# systemctl enable k8s-kms-plugin
# systemctl start k8s-kms-plugin

Verify that the K8S-KMS-Plugin service is running.

# systemctl status k8s-kms-plugin -l --no-pager

Verify that the socket file is created under the /tmp/run directory specified in k8s-kms-plugin.yaml.

# ls -ltr /tmp/run

Create the encryption configuration file encryption-config.yaml, once K8S-KMS-Plugin is ready to use as an encryption provider, on the master node where the API server is running:

---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: hsm-plugin
          endpoint: unix:///tmp/run/hsm-plugin-server.sock
          timeout: 3s
      - identity: {}

Save the encryption configuration file encryption-config.yaml at any location on the master host. For example:

/etc/kubernetes/kms/encryption-config.yaml

Open the kube-apiserver.yaml file from the static pod manifest location and edit it to add the --encryption-provider-config flag to the kube-apiserver command list. The flag must point to the encryption-config.yaml file.

spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/kms/encryption-config.yaml
        - --encryption-provider-config-automatic-reload=true
        - --advertise-address=10.164.75.18
        - --allow-privileged=true
        - --authorization-mode=Node,RBAC
        ...

To allow the kube-apiserver pod to communicate with hsm-plugin-server.sock, add access to the directories where the encryption-config.yaml file and the Unix socket are located. Add mount points in kube-apiserver.yaml as shown below, ensuring correct indentation:

...
    volumeMounts:
      - mountPath: /etc/kubernetes/kms
        name: kms
        readOnly: true
      - mountPath: /tmp/run
        name: socket
...
volumes:
  - hostPath:
      path: /etc/kubernetes/kms
      type: DirectoryOrCreate
    name: kms
  - hostPath:
      path: /tmp/run
      type: DirectoryOrCreate
    name: socket
...

Save and close the kube-apiserver.yaml file. The API server restarts automatically when you save the changes. Ensure that your cluster is back online without any failure and the API server is READY and Running.

# kubectl get pods --all-namespaces

Confirm that the API server is configured successfully to use hsm-plugin-server.sock and that the connection is established by running the following command on the master node:

# ss -a --unix -p | grep hsm-plugin-server.sock

Create a new Secret called mysecret in the default namespace.

# kubectl create secret generic mysecret -n default --from-literal=mykey=mys3cr3t

The encrypted Secret is saved in etcd.

Read the Secret out of etcd using the etcdctl command line. The command is listed below with parameter values; change all parameter values to match your environment.

# alias etcdctl3="ETCDCTL_API=3 /run/containerd/io.containerd.runtime.v2.task/k8s.io/8e56d11c88ec8c5cb0add95c8add48343ec6f4bbf1a30a82c5c0e85b7abdb540/rootfs/usr/local/bin/etcdctl --endpoints="https://127.0.0.1:2379" --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key --cacert=/etc/kubernetes/pki/etcd/ca.crt"

# etcdctl3 get /registry/secrets/default/mysecret

You will see the kms:v2:hsm-plugin identifier and the encrypted value, as shown below:

Run the following command to confirm that the stored Secret is prefixed with k8s:enc:kms:v2:hsm-plugin, which indicates that the KMS provider has encrypted the resulting data.

# etcdctl3 get /registry/secrets/default/mysecret | hexdump -C

Confirm that the Secret decrypts correctly when retrieved via the API server.

# kubectl get secret mysecret -n default -o yaml

Run the following command to encrypt all pre-existing Secrets. The command reads all Secrets and re-encrypts them using the KMS provider.

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -

If an error occurs due to a conflicting write, retry the command. For larger clusters, it is recommended to subdivide the Secrets by namespace or script an update. All Secrets are now encrypted using the key generated on the Luna HSM via K8S-KMS-Plugin. The Secrets are decrypted automatically by K8S-KMS-Plugin when called by the API server.

Once all Secrets are encrypted, remove the identity provider to prevent the API server from honoring unencrypted data. Edit /etc/kubernetes/kms/encryption-config.yaml, remove the identity provider at the end, and save the file.

---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: hsm-plugin
          endpoint: unix:///tmp/run/hsm-plugin-server.sock
          timeout: 3s

Edit the encryption configuration file encryption-config.yaml on the master host. Add the KMS provider as the first entry in the configuration file, as shown in the example below:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: hsm-plugin
          endpoint: unix:///tmp/run/hsm-plugin-server.sock
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64 ENCODED SECRET>

Where key1 is the name of your key and <BASE64 ENCODED SECRET> is the actual key of the native encryption provider. Ensure that the Unix socket path matches the actual socket path mounted in the kube-apiserver manifest YAML.

Restart all kube-apiserver processes.

Run the following command to force all Secrets to be re-encrypted using the KMS provider.

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Generate a new RSA or AES key on the Luna HSM, following the steps in Generate Key Encryption Key on Luna HSM.

Open the environment configuration file that the K8S-KMS-Plugin service reads at startup.

# vi /etc/kms-plugin/k8s-kms-plugin.yaml

Configure the key rotation settings. Add the newly generated KEK reference to the serve section and the existing KEK reference to the rotation section. The serve section specifies the new KEK used for encryption, while the rotation section specifies the existing KEK used to decrypt data encrypted with the previous key. Adjust the values to match your environment.

k8s-kms-plugin:
  log-level: "trace"
  serve:
    socket: "/tmp/run/hsm-plugin-server.sock"
    provider: "luna"
    p11-lib: "/usr/safenet/lunaclient/lib/libCryptoki2_64.so"
    p11-label: "TPA-HA"
    p11-pin: "userpin1"
    p11-key-label: "AES01"  # Newly generated KEK label
    algorithm: "aes-gcm"
  # Key rotation settings
  rotation:
    old-provider: "luna"
    old-p11-lib: "/usr/safenet/lunaclient/lib/libCryptoki2_64.so"
    old-p11-label: "TPA-HA"
    old-p11-pin: "userpin1"
    old-p11-key-label: "rsa01"  # Existing or old KEK
    old-algorithm: "rsa-oaep"

Update the K8S-KMS-Plugin service and add the rotation subcommand to the existing ExecStart command.

# vi /etc/systemd/system/k8s-kms-plugin.service

...
ExecStart=/usr/bin/k8s-kms-plugin serve rotation --config /etc/kms-plugin/k8s-kms-plugin.yaml
...

Restart the K8S-KMS-Plugin service.

# systemctl daemon-reload
# systemctl restart k8s-kms-plugin

Verify that the K8S-KMS-Plugin service is running.

# systemctl status k8s-kms-plugin -l --no-pager

Restart all kube-apiserver processes to bring the newly configured KEK into effect.

Optionally, run the following command to force all Secrets to be re-encrypted using the KMS provider.

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Update the K8S-KMS-Plugin configuration and service files as follows:

Comment out or remove the rotation section of /etc/kms-plugin/k8s-kms-plugin.yaml:

k8s-kms-plugin:
  log-level: "trace"
  serve:
    socket: "/tmp/run/hsm-plugin-server.sock"
    provider: "luna"
    p11-lib: "/usr/safenet/lunaclient/lib/libCryptoki2_64.so"
    p11-label: "TPA-HA"
    p11-pin: "userpin1"
    p11-key-label: "AES01"  # Newly generated KEK label
    algorithm: "aes-gcm"
  # Key rotation settings
  #rotation:
  #  old-provider: "luna"
  #  old-p11-lib: "/usr/safenet/lunaclient/lib/libCryptoki2_64.so"
  #  old-p11-label: "TPA-HA"
  #  old-p11-pin: "userpin1"
  #  old-p11-key-label: "rsa01"  # Existing or old KEK
  #  old-algorithm: "rsa-oaep"

Remove the rotation subcommand from the ExecStart command in /etc/systemd/system/k8s-kms-plugin.service:

...
ExecStart=/usr/bin/k8s-kms-plugin serve --config /etc/kms-plugin/k8s-kms-plugin.yaml
...

Restart the K8S-KMS-Plugin service.

# systemctl daemon-reload
# systemctl restart k8s-kms-plugin