Troubleshooting
This section contains the following troubleshooting topics:
Inspecting Deployment
Deployment of CTE for Kubernetes typically uses a single controller pod and at least one worker pod per Kubernetes node. Inspect the deployment using kubectl get
commands to access basic output.
Example
Get the CTE for Kubernetes pod names that were deployed on the Kubernetes cluster, type:
NAMESPACE="kube-system"
kubectl get pods -n ${NAMESPACE} -o wide | grep cte-csi
Response
cte-csi-controller-0 2/2 Running 0 25h 192.168.77.182 ub20-work1 <none> <none>
cte-csi-node-28srz 4/4 Running 0 25h 192.168.77.181 ub20-work1 <none> <none>
cte-csi-node-cjzzn 4/4 Running 0 25h 192.168.76.68 ub20-master <none> <none>
Getting CTE for Kubernetes Agent version
The version of CTE for Kubernetes that you are running displays in the log files for each pod:
NAMESPACE="kube-system"
for podName in `kubectl get pods -n ${NAMESPACE} | grep cte-csi-[node,controller] | cut -d " " -f1`; do
echo -n "${podName}: "
kubectl logs -n ${NAMESPACE} ${podName} cte-csi | grep "Version:"
done
Response
cte-csi-controller-7d7d8bd46b-wbffd: I0308 17:25:13.608929 1 cte.go:139] Version: 1.2.0.123
cte-csi-node-8qx5q: I0308 17:25:11.643673 2218968 cte.go:139] Version: 1.2.0.123
cte-csi-node-h8q8q: I0308 17:25:13.465795 2205645 cte.go:139] Version: 1.2.0.123
Inspecting Events
CTE for Kubernetes relies on Kubernetes event infrastructure for diagnostics of problems with a CTE for Kubernetes volume. Any errors in attaching a volume to a pod will display in the event logs. Look at all of the events in a Kubernetes namespace by typing:
NAMESPACE="kube-system"
kubectl get event -n ${NAMESPACE}
To target events of a specific pod, type:
PODNAME="my_application_pod"
NAMESPACE="kube-system"
kubectl get event -n ${NAMESPACE} --field-selector involvedObject.name=${PODNAME}
Inspecting CTE for Kubernetes logs
Examining logs for CTE for Kubernetes could offer supplementary insights that might not be conveyed through Kubernetes Events.
Deployment of CTE for Kubernetes is split between two different types of pods:
-
Controller Server: Manages the dynamic provisioning of CTE for Kubernetes persistent volumes. Denoted by a
cte-csi-controller-X
pod name. -
Node server: Manages attaching CTE for Kubernetes volumes to pods. Denoted by a
cte-csi-node-XXXXX
pod name.
Inspecting the Controller Server
Inspect the logs for the controller server to debug CTE for Kubernetes persistent volume provisioning issues:
NAMESPACE="kube-system"
kubectl logs -n ${NAMESPACE} cte-csi-controller-0 cte-csi
Inspecting the Node Server
The logs for the CTE for Kubernetes pods are distributed across two containers:
-
The
cte-csi container
contains logs relevant to CTE for Kubernetes activity, encompassing details like volume mounting and registration -
The
cte-agent-logs
for the CTE encryption engine agent can be found in the container.
Inspecting the Node Server requires first identifying which Kubernetes node the application pod is scheduled on. Once that node has been identified, find the CTE for Kubernetes node server running on that node.
After you obtain the pod name, use the two relevant kubectl logs
commands to view the logs, type:
NAMESPACE="kube-system"
kubectl logs -n ${NAMESPACE} cte-csi-node-XXXXX cte-csi
kubectl logs -n ${NAMESPACE} cte-csi-node-XXXXX cte-agent-logs
Problems with Registration
CTE for Kubernetes automatically registers to CipherTrust Manager based on demand for volumes on a node. Failure to register is typically due to the registration token being either invalid or the token has no more client capacity. These types of errors are reported back through Kubernetes so analyzing the log files of the troubled pod reveals the registration failure message seen by the agent.
Troubleshooting Trusted Pods failures
The following are two examples of trusted pod failures:
Example 1
In cte-csi-node
logs, the following error indicates that a running pod digest was not found in any signature set attached to a security policy.
E0310 06:41:16.614410 1687839 server.go:106] GRPC error: rpc error: code = Internal desc = Pod did not pass attestation checks: rpc error: code = Internal desc = Found no signature set for container ubuntu with digest 2adf22367284330af9f832ffefb717c78239f6251d9d0f58de50b86229ed1427
Example 2
In cte-csi-node
logs, the following error indicates that running pod digests cannot be matched to the same signature set. There were two containers (ubuntu and ubuntu2) with digests included in different signature sets. Partial matches are displayed to help with troubleshooting.
E0407 09:07:33.393609 1492044 server.go:106] GRPC error: rpc error: code = Internal desc = Pod did not pass attestation checks: rpc error: code = Internal desc = Pod attestation failed! Unable to match all pod digests to same signature set. Partial matches:
Signature set policy-sigset2: [ubuntu:8ae9bafbb64f63a50caab98fd3a5e37b3eb837a3e0780b78e5218e63193961f9]
Signature set policy-sigset3: [ubuntu2:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a]
Backing up Databases after Encryption
After encrypting a database, CipherTrust Transparent Encryption cannot make a backup of the database. Both scheduled and manual backup fail. The problem was the user's policy. A policy used in this scenario must follow a few rules.
With a CBC_CS1 key, a guarded file is modified to have a 4096 byte header holding key information. When an Apply Key effect is specified, the CipherTrust Transparent Encryption code adjusts the length and file offset for this header. Without an Apply Key effect, the size and access of the offset include the CBC_CS1 header.
Thales recommends that you modify the first rule of your policy. Remove the action entry for f_rd_att
from the first rule and add a new rule before it:
**action**: f_rd_att
**effect**: Permit, Apply Key
Policy processing starts with the first rule and continues until a matching rule is found. The effect for the matching rule is then applied.
For the f_rd_att
action, this results in the Secfs code including the CBC_CS1 key header and adjusts the file size value. Without the Apply Key effect, the file size includes the CBC_CS1 header size and the file appears as 4096 bytes larger than its real size.
FSGroupID is not working with NFS shared storage volume
The fsgroup
ID Security Context option allows an administrator to change the permissions of volumes before a pod starts. Some users have found that when adding the fsgroup
ID to the SecurityContext section in the pod yaml file, it does not work as expected in the NFS storage environment.
The reason is that the fsgroup
ID Security Context option is not supported with NFS volumes. It is only supported with local storage. This is a limitation from Kubernetes and not an issue with the CTE-U fuse driver.
- See Configure a Security Context for a Pod or Container in the Kubernetes documentation for more information.