High Availability

This section explains the procedure to integrate the CipherTrust Manager present in a cluster with Spectrum Scale.

Integration Steps

In a high availability setup, primary and backup CipherTrust Manager server(s) can be configured in the RKM.conf file.

• The kmipServerUri attribute represents the DNS name or IP address of the CipherTrust Manager and the KMIP SSL port.

• The backup server(s) should be listed in the kmipServerUri2, kmipServerUri3, etc.

  Note

  • If you're using External CA, Local CA for Automatic Server Certificate Generation should be set to Turn off auto generation from Local CA.

  • In case of External CA , update the certificate chain (server cert , CA , server key) in the KMIP interface of the backup server, same as of the primary server.

  • For high availability (HA), it is recommended to add multiple backup servers, that is, multiple kmipServerUri entries. You can have up to five backup key servers.

The following example has the primary key server and a secondary key server.

1. Add the following RKM (Remote Key Managment) stanza to the RKM.conf file (created in the Pre-Integration section). This stanza contains the information required to connect to the CipherTrust Manager server.

   RKM_Stanza {
   type = KMIP
   kmipServerUri = tls://10.164.x.xxx:5696
   kmipServerUri2 = tls://10.164.x.abc:5696
   keyStore = /var/mmfs/etc/RKMcerts/client.p12
   passphrase = abcd
   clientCertLabel = client
   }
   

   Brief description of fields contained in the stanza:

   • RKM_Stanza: Name of the RKM Stanza. It can be set as desired, "RKM_Stanza" here is just an example name for the stanza.

   • type: Must be KMIP.

   • kmipServerUri: IP addresses or the DNS names of the CipherTrust Manager and their SSL ports.

     • For high availability (HA), multiple kmipServerUri entries may be added.

     • In an event when a CipherTrust Manager fails or is inaccessible, then request is sent to the other nodes specified here.

   • keyStore: The name and path of the client.p12 file (for example: /var/mmfs/etc/RKMcerts/client.p12).
     

   • passphrase: The password of the client keystore and client certificate. This is the same password which you specified while creating client.p12 file.
     

   • clientCertLabel: The label of the client certificate in the client keystore. This is the same label which you specified while creating client.p12 file.
     

2. Install the encryption policy on the file system using the following command.

   /usr/lpp/mmfs/bin/mmchpolicy fs0 hsm62
   

Verifying Integration

To verify the integration performed using High Availability, follow the below steps:

1. Create a Key Export call in the primary CipherTrust Manager of the cluster.

2. Shut down the CipherTrust Manager.

3. Clear the key cache on the IBM Spectrum Scale client.

   /usr/lpp/mmfs/bin/tsctl encKeyCachePurge 'cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client'
   

   Note

   • cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56 is the ID for the key created in the CipherTrust Manager.

   • client is the RKM stanza ID.

   Both values can be retrieved by running the command: mmlspolicy fs0 -L

   For example:

   [root@ss1 etc]# /usr/lpp/mmfs/bin/mmlspolicy fs0 -L
   RULE 'p1' SET POOL 'system' /* one placement rule is required at all times */
   RULE 'Encrypt all files in file system with rule E1'
   SET ENCRYPTION 'E1'
   WHERE NAME LIKE '%'
   RULE 'rule1' ENCRYPTION 'E1' IS
   ALGO 'DEFAULTNISTSP800131A'
   KEYS('cd1929c52e4b2e5a8047d6e6815cf41df8d4d664bb78b85991ecce5bce56:client')
   

4. Access an existing encrypted file or try to create a new encrypted file. The Spectrum Scale daemon log will show a warning message that indicates its inability to retrieve the encryption key from the primary key server:

   2020-04-07_11:51:19.637-0400: [W] The key server 10.164.x.xxx (port 5696) had a failure and will be quarantined for 1 minute(s).
   

   Spectrum Scale will try the second key server in the list, and the I/O operation should be successful if the secondary/backup server is able to serve the encryption key.

5. Check CipherTrust Manager activity logs or records for Key Export calls.

   Note

   If integration is successful, then KMIP encryption operation is processed by secondary server present in the cluster. That means the Key Export calls would be created in secondary server activity logs if the primary is down.

6. If the Key Export calls are created on the secondary CipherTrust Manager present in the cluster, then the integration is successful.