Troubleshooting

This section contains the following troubleshooting topics:

• Inspecting Deployment

• Getting CTE for Kubernetes Agent version

• Inspecting Events

• Inspecting CTE for Kubernetes logs

• Inspecting the Controller Server

• Inspecting the Node Server

• Problems with Registration

• Troubleshooting Trusted Pods failures

• Backing up Databases after Encryption

Inspecting Deployment

Deployment of CTE for Kubernetes typically uses a single controller pod and at least one worker pod per Kubernetes node. Inspect the deployment using kubectl get commands to access basic output.

Example

Get the CTE for Kubernetes pod names that were deployed on the Kubernetes cluster, type:

NAMESPACE="kube-system"
kubectl get pods -n ${NAMESPACE} -o wide | grep cte-csi

Response

cte-csi-controller-0      2/2     Running   0      25h    192.168.77.182   ub20-work1    <none>   <none>
cte-csi-node-28srz        4/4     Running   0      25h    192.168.77.181   ub20-work1    <none>   <none>
cte-csi-node-cjzzn        4/4     Running   0      25h    192.168.76.68    ub20-master   <none>   <none>

Getting CTE for Kubernetes Agent version

The version of CTE for Kubernetes that you are running displays in the log files for each pod:

NAMESPACE="kube-system"
for podName in `kubectl get pods -n ${NAMESPACE} | grep cte-csi-[node,controller] | cut -d " " -f1`; do
    echo -n "${podName}:  "
    kubectl logs -n ${NAMESPACE} ${podName} cte-csi | grep "Version:"
done

Response

cte-csi-controller-7d7d8bd46b-wbffd:  I0308 17:25:13.608929 1 cte.go:139] Version: 1.2.0.123
cte-csi-node-8qx5q:  I0308 17:25:11.643673 2218968 cte.go:139] Version: 1.2.0.123
cte-csi-node-h8q8q:  I0308 17:25:13.465795 2205645 cte.go:139] Version: 1.2.0.123

Inspecting Events

CTE for Kubernetes relies on Kubernetes event infrastructure for diagnostics of problems with a CTE for Kubernetes volume. Any errors in attaching a volume to a pod will display in the event logs. Look at all of the events in a Kubernetes namespace by typing:

NAMESPACE="kube-system"
kubectl get event -n ${NAMESPACE}

To target events of a specific pod, type:

PODNAME="my_application_pod"
NAMESPACE="kube-system"
kubectl get event -n ${NAMESPACE} --field-selector involvedObject.name=${PODNAME}

Inspecting CTE for Kubernetes logs

Examining logs for CTE for Kubernetes could offer supplementary insights that might not be conveyed through Kubernetes Events.

Deployment of CTE for Kubernetes is split between two different types of pods:

• Controller Server: Manages the dynamic provisioning of CTE for Kubernetes persistent volumes. Denoted by a cte-csi-controller-X pod name.

• Node server: Manages attaching CTE for Kubernetes volumes to pods. Denoted by a cte-csi-node-XXXXX pod name.

Inspecting the Controller Server

Inspect the logs for the controller server to debug CTE for Kubernetes persistent volume provisioning issues:

NAMESPACE="kube-system"
kubectl logs -n ${NAMESPACE} cte-csi-controller-0 cte-csi

Inspecting the Node Server

The logs for the CTE for Kubernetes pods are distributed across two containers:

• The cte-csi container contains logs relevant to CTE for Kubernetes activity, encompassing details like volume mounting and registration

• The cte-agent-logs for the CTE encryption engine agent can be found in the container.

Inspecting the Node Server requires first identifying which Kubernetes node the application pod is scheduled on. Once that node has been identified, find the CTE for Kubernetes node server running on that node.

After you obtain the pod name, use the two relevant kubectl logs commands to view the logs, type:

NAMESPACE="kube-system"
kubectl logs -n ${NAMESPACE} cte-csi-node-XXXXX cte-csi
kubectl logs -n ${NAMESPACE} cte-csi-node-XXXXX cte-agent-logs

Problems with Registration

CTE for Kubernetes automatically registers to CipherTrust Manager based on demand for volumes on a node. Failure to register is typically due to the registration token being either invalid or the token has no more client capacity. These types of errors are reported back through Kubernetes so analyzing the log files of the troubled pod reveals the registration failure message seen by the agent.

Troubleshooting Trusted Pods failures

The following are two examples of trusted pod failures:

Example 1

In cte-csi-node logs, the following error indicates that a running pod digest was not found in any signature set attached to a security policy.

E0310 06:41:16.614410 1687839 server.go:106] GRPC error: rpc error: code = Internal desc = Pod did not pass attestation checks: rpc error: code = Internal desc = Found no signature set for container ubuntu with digest 2adf22367284330af9f832ffefb717c78239f6251d9d0f58de50b86229ed1427

Example 2

In cte-csi-node logs, the following error indicates that running pod digests cannot be matched to the same signature set. There were two containers (ubuntu and ubuntu2) with digests included in different signature sets. Partial matches are displayed to help with troubleshooting.

E0407 09:07:33.393609 1492044 server.go:106] GRPC error: rpc error: code = Internal desc = Pod did not pass attestation checks: rpc error: code = Internal desc = Pod attestation failed! Unable to match all pod digests to same signature set. Partial matches:
Signature set policy-sigset2: [ubuntu:8ae9bafbb64f63a50caab98fd3a5e37b3eb837a3e0780b78e5218e63193961f9]
Signature set policy-sigset3: [ubuntu2:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a]

Backing up Databases after Encryption

After encrypting a database, CipherTrust Transparent Encryption cannot make a backup of the database. Both scheduled and manual backup fail. The problem was the user's policy. A policy used in this scenario must follow a few rules.

With a CBC_CS1 key, a guarded file is modified to have a 4096 byte header holding key information. When an Apply Key effect is specified, the CipherTrust Transparent Encryption code adjusts the length and file offset for this header. Without an Apply Key effect, the size and access of the offset include the CBC_CS1 header.

Thales recommends that you modify the first rule of your policy. Remove the action entry for f_rd_att from the first rule and add a new rule before it:

**action**: f_rd_att

**effect**: Permit, Apply Key

Policy processing starts with the first rule and continues until a matching rule is found. The effect for the matching rule is then applied.

For the f_rd_att action, this results in the secfs code including the CBC_CS1 key header and adjusts the file size value. Without the Apply Key effect, the file size includes the CBC_CS1 header size and the file appears as 4096 bytes larger than its real size.