Appendix
Troubleshooting
You may face following errors while performing the integration steps mentioned in previous sections. The document lists errors from both appliances separately:
IBM Spectrum Scale Errors
| Error | Troubleshooting |
|---|---|
| If the IBM Spectrum Scale client is not able to contact the key server when accessing an encrypted file, the following error message can be found in the daemon log: 2020-04-23_13:06:47.564-0400: [E] Key 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client' could not be fetched. The TCP connection with the RKM could not be established. | Check the connectivity between the client and the key server. |
| The following error message is printed in the daemon log when a bad certificate is detected: 2020-04-22_18:13:06.623-0400: [E] Key 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client' could not be fetched. Bad certificate. | Check and ensure the client and server certificates are valid. |
| The following error message indicates that the /var/mmfs/etc/RKM.conf file is missing the RKM stanza "RKM3": 2020-04-28_12:00:23.459-0400: [E] Key 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client' could not be fetched. The specified RKM ID does not exist; check the RKM.conf settings. | Ensure that /var/mmfs/etc/RKM.conf on all nodes contains the required RKM stanza. |
| The following error message indicates that the client key store file specified in the /var/mmfs/etc/RKM.conf file is missing: 2020-05-01_17:20:22.972-0400: [E] Error while validating policy 'for file system dsm64': rc=778: While parsing file '/var/mmfs/etc/RKM.conf':[E] Could not open the key store file ('/var/mmfs/etc/RKMcerts/dsm64Client.p12' was specified). | Check the /var/mmfs/etc/RKM.conf file and ensure that the correct client keystore file is specified in the keyStore attribute. Verify that the client keystore (.p12) file is present on all nodes in the cluster. |
| The following error message indicates that the permissions on the client keystore file are incorrect. Access to this file should be granted to root only: 2020-05-01_17:25:24.173-0400: [E] Error while validating policy 'for file system dsm64': rc=778: While parsing file '/var/mmfs/etc/RKM.conf':[E] The keyStore permissions are incorrect for /var/mmfs/etc/RKMcerts/dsm64Client.p12. Access should be only granted to root, and no execute permission is allowed for the file | Check and fix the permissions on the client keystore file. |
| If RKM.conf file is not copied to all nodes, following error may occur: [root@ss1 etc]# /usr/lpp/mmfs/bin/mmchpolicy fs0 hsm62 Validated policy 'hsm62': Parsed 3 policy rules. The RKM identifier specified for key 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client' does not exist; check the RKM.conf file. Make sure the RKM.conf and keystore(s) files are copied to all nodes in the cluster. | Copy the RKM.conf file on all nodes. |
| Setting incorrect or insufficient permissions for RKM.conf file may cause the following error: [root@ss1 etc]# /usr/lpp/mmfs/bin/mmchpolicy fs0 hsm62 [E] Error while validating policy 'hsm62': rc=778:[E] Incorrect permissions for the configuration file /var/mmfs/etc/RKM.conf on node ss2. mmchpolicy: Command failed. Examine previous error messages to determine cause. | Set correct permissions for RKM.conf file |
| If keystore file is missing from /var/mmfs/etc/RKMcerts folder, following error may occur: [root@ss1 etc]# /usr/lpp/mmfs/bin/mmchpolicy fs0 hsm62 [E] Error while validating policy 'hsm62': rc=778: While parsing file '/var/mmfs/etc/RKM.conf':[E] Could not open the key store file ('/var/mmfs/etc/RKMcerts/client.p12' was specified). mmchpolicy: Command failed. Examine previous error messages to determine cause. | Make sure that keystore file is in /var/mmfs/etc/RKMcerts |
CipherTrust Manager Errors
| Error | Troubleshooting |
|---|---|
| If the username specified while creating a key is not same as the CN (Common Name) specified in the client certificate, following error may occur: [root@ss1 etc]# cp /root/test.txt /fs0 cp: cannot create regular file '/fs0/test.txt': Operation not permitted [root@ss1 etc]# vim /var/adm/ras/mmfs.log.latest 2020-05-20_08:25:47.961-0400: [E] Unable to create encrypted file test.txt (inode 86532, fileset 0, file system fs0). 2020-05-20_08:25:47.962-0400: [E] Key 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client' could not be fetched (RKM reported error -1256). "/var/adm/ras/mmfs.log.latest" 537L, 49635C | Specify the Username same as the CN (Common Name) specified while creating the client certificate. |
| If mode of the KMIP Interface is selected as TLS, Verify client cert, user must supply password, following error may occur. Error [root@ss1 etc]# cp /root/test2.txt /fs0 cp: cannot create regular file '/fs0/test2.txt': Operation not permitted [root@ss1 etc]# vim /var/adm/ras/mmfs.log.latest 2020-05-20_08:34:13.693-0400: [W] The key server 10.164.10.118 (port 9889) had a failure and will be quarantined for 1 minute(s). 2020-05-20_08:34:13.924-0400: [W] The key server 10.164.10.123 (port 9889) had a failure and will be quarantined for 1 minute(s). 2020-05-20_08:34:13.924-0400: [E] Unable to create encrypted file test2.txt (inode 86541, fileset 0, file system fs0). 2020-05-20_08:34:13.924-0400: [E] Key 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client' could not be fetched (RKM reported error -115). | IBM Spectrum Scale does not support this setting. Choose one of the other two options: • Optional • Not Used |
| If you are using external CA and primary server fails to serve the key | In KMIP interface, Local CA for Automatic Server Certificate Generation should be set to Turn off auto generation from Local CA. |
