System Upgrade/Downgrade

You can upgrade/downgrade your CipherTrust Manager firmware to and from 2.11.x by securely downloading and applying a new/older system archive file.

Specific upgrade steps vary depending on whether the CipherTrust Manager is clustered or unclustered.

Downgrade is only supported on unclustered appliances.

System Upgrade for an Unclustered Appliance

Refer to Cluster Upgrade for details on upgrading a CipherTrust Manager which is part of a cluster of devices.

For 2.11.x, we tested upgrade from the latest patch versions of 2.10.x, 2.9.x, and 2.8.x. As well, we tested upgrades from a lower 2.11.x patch version, such as 2.11.2, to a higher patch version, such as 2.11.5.

You require ksadmin level access with an SSH key.

1. Obtain the signed archive file for the upgrade from the Support Portal. The file has the format ks_upgrade_<major.minor.patch+build_number>.tar.gz.gpg.

2. On CipherTrust Manager create and download a backup with corresponding backup key, in case there are any problems.

   Note

   Upgrades keep all the data and may migrate the data and configuration. Therefore, as a precaution, it is recommended to take a backup before upgrading.
   Consult the backups page for details on forward compatibility for backups. Restoring a newer backup to an older version is never supported.

3. Transfer the archive file to the CipherTrust Manager. You require the private SSH key associated with the ksadmin account. The command and supported private key format depends on the operating system you are transferring the archive file from.

   Source Operating System	Example Command Syntax	Required Private Key Format
   Linux	scp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	OpenSSH format
   Windows	pscp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	PuTTY PPK format

4. ssh into the CipherTrust Manager as ksadmin and ensure there is sufficient disk space available. The required disk space is 5 times the size of the upgrade file. For example, if the upgrade file is 5 GB, 25 GB of free space is required. Use df -h / to view available space.

5. Run the following command:

   sudo /opt/keysecure/ks_upgrade.sh -f <archive_file_path>

   Here, <archive_file_path> specifies the CipherTrust Manager path to the signed archive file.

   The signature of the archive file is verified and the upgrade is applied.

6. Reboot the appliance when prompted.

7. Ensure the CipherTrust Manager services have started. From the ksadmin session, run systemctl status keysecure. Alternatively, you can visit the CipherTrust Manager web console or attempt to connect with the ksctl CLI.

8. If you need to verify the new version, look for the version displayed in CipherTrust Manager banner in the SSH session on reboot, as shown in the screenshot. Alternatively, log in the web console UI and click the information icon at the top right to display version information.

   

9. Re-download and re-install ksctl to access commands and settings introduced in the new CipherTrust Manager software version.

   Note

   Previous versions of ksctl still work with new CipherTrust Manager software for existing features and settings.

System Downgrade

An unclustered CipherTrust Manager 2.11.x can be downgraded to 2.10.x. For release-specific upgrade/downgrade information, refer to the release notes for your release.

Downgrades perform a CipherTrust Manager reset, which wipes all CipherTrust Manager data except the backup files that already exist.

As well, the PCI HSM drivers on k570 models, and base operating system packages are not changed during downgrade.

To downgrade your CipherTrust Manager

1. SSH into the CipherTrust Manager as "ksadmin".

2. Downgrade the CipherTrust Manager:

   $ sudo /opt/keysecure/ks_downgrade.sh -f <~/filename>
   

Usage: ks_downgrade.sh -f <FILE> [-o]

*  `-f`: Path to the signed CipherTrust Manager installer file.

*  `-o`: Clustered node cannot be downgraded. Use this flag to override this behavior.

Cluster Upgrade

A cluster can be upgraded in three ways:

• Online In-place Cluster Upgrade

• Offline In-place Cluster Upgrade

• Using the Cluster Remove/Rebuild Method

The following table indicates the differences between the three ways:

Prerequisites

• You require ksadmin level access with an SSH key.

• Obtain the signed archive file for the upgrade from the Support Portal. The file has the format ks_upgrade_<major.minor.patch+build_number>.tar.gz.gpg.

Online In-place Cluster Upgrade

A cluster can be upgraded in-place. The upgrade is generally limited to one minor version at a time, for example, from 2.8.0 to 2.9.0; from 2.9.0 to 2.10.0; or from 2.10.0 to 2.11.1. As well, you can use cluster upgrade to move to any higher patch within the same minor version. For example, 2.11.1 to 2.11.5 is a valid increment.

Be aware of the following considerations when performing an in-place cluster upgrade.

• The node being upgraded will be inaccessible during the upgrade. This may be as long at 10 or more minutes. Clients must be able to handle this outage.

• There will be a brief period of time (under 30 seconds) where the database will be locked while upgrading the first node. This affects all nodes at the same time, and some nodes may give error responses during this time.

• All nodes in the cluster should be upgraded as soon as possible - nodes running different version of the firmware will behave differently, potentially causing problems with applications.

To perform an online in-place cluster upgrade

1. Before doing any upgrade operation, ensure that you have a backup, and that you have downloaded the backup and associated backup key.

2. Ensure all nodes in the cluster are up and operating normally. Resolve any issues (like removing any obsolete nodes) before performing the upgrade.

3. Perform a system upgrade on each node, one at a time.

   1. Transfer the archive file to the CipherTrust Manager. You require the private SSH key associated with the ksadmin account. The command and supported private key format depends on the operating system you are transferring the archive file from.

      Source Operating System	Example Command Syntax	Required Private Key Format
      Linux	scp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	OpenSSH format
      Windows	pscp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	PuTTY PPK format

   2. SSH into the CipherTrust Manager as ksadmin and ensure there sufficient disk space available. The required disk space is 5 times the size of the upgrade file. For example, if the upgrade file is 5 GB, 25 GB of free space is required. Use df -h / to view available space.

   3. Run the following command to upgrade the device.

      $ sudo /opt/keysecure/ks_upgrade.sh -f <archive_file_path>
      

      Here, <archive_file_path> specifies the CipherTrust Manager path to the signed archive file.

      The signature of the archive file is verified and the upgrade is applied.

   4. Reboot the appliance as prompted.

   5. Ensure the upgrade of each node is complete and that the node is operating normally, before proceeding to the next node.

      From the ksadmin session, run systemctl status keysecure to ensure that the CipherTrust Manager services have started. Alternatively, you can visit the CipherTrust Manager web console or attempt to connect with the ksctl CLI.

      Note

      When updating the first node in a cluster, the cluster nodes may briefly experience slower than usual response times. This occurs because the shared database schema for the cluster is updated with the first node.

   6. If you need to verify the new version, look for the version displayed in CipherTrust Manager banner in the SSH session on reboot, as shown in the screenshot. Alternatively, log in the web console UI and click the information icon at the top right to display version information.

      

4. Repeat step 3 for each minor version.

Offline In-Place Cluster Upgrade

If your target CipherTrust Manager version is a Long Term Support (LTS) release, you can leave the cluster intact with an offline upgrade. Consult the release notes of the target version for supported upgrade paths. For 2.11.x, offline upgrade is available from 2.10.x. As well, we tested upgrades from a lower 2.11.x patch version, such as 2.11.2, to a higher patch version, such as 2.11.5.

Prerequisites for Offline Cluster Upgrade

• You need to login as ksadmin with an SSH key from your client workstation to each node individually to accept the host fingerprint.

  Note

  You need to be able to both use SSH and run the cmupdate tool in the Windows command prompt, as is supported in Windows 10 and higher. You cannot use a separate SSH client utility such as PuTTY.

• Ensure that you have a backup, and that you have downloaded the backup and associated backup key.

• Ensure all nodes in the cluster are up and operating normally. Resolve any issues (like removing any obsolete nodes) before performing the upgrade.

To perform an offline in-place cluster upgrade

1. Obtain the cmupdate utility for your client workstation operating system and install it.

   The following cmupdate utilities are available:

   • cmupdate-linux-amd64 for 64-bit Linux system

   • cmupdate-windows-amd64.exe for 64-bit Windows system.

     Note

     Windows 10 is the minimum supported version for the client workstation.

   • cmupdate-darwin-amd64 for 64-bit macOS system with Intel processor.

     Note

     For some versions of macOS, a prompt appears indicating that macOS cannot verify the developer. You need to dismiss the prompt and navigate into Security & Privacy settings to manually allow cmupdate.

2. Rename the utility to cmupdate, without the OS-specific suffix.

3. Run cmupdate in a terminal or command prompt program with full privileges to confirm it is set up properly.

   If successful, text similar to the following is displayed:

   Command line tool for upgrading clustered and standalone CipherTrust Managers
   
   Usage:
     cmupdate [command]
   
   Available Commands:
     completion  Generate the autocompletion script for the specified shell
     help        Help about any command
     prepare     Prepare the CM(s) for a system upgrade
     upgrade     Perform a system upgrade on a standalone CM or multiple nodes in a cluster
     version     Output tool version
   
   Flags:
     -h, --help      help for cmupdate
     -v, --verbose   Provide verbose output while executing command (optional)
   
   Use "cmupdate [command] --help" for more information about a command.
   

4. Run the cmupdate prepare command to check if the cluster is ready for upgrade and to upload the upgrade file. You require the upgrade file, the SSH keys of each CipherTrust Manager node, and the IP address or hostname for at least one node.

   1. Run the command.

      cmupdate prepare --identities <ssh_private_key_1,ssh_private_key_2 ...> --nodes <hostname_or_IP_address_of_a_node> --file <upgrade_file_path> --all
      

   2. You are asked to confirm the hostnames or IP addresses of every node in the cluster.

      The following hosts are part of the cluster and will be upgraded: 1.1.1.1,1.1.1.2 Continue (y/n)? y
      

      The prepare command outputs to a log file in your local directory.

      Note

      The upgrade file is uploaded to all nodes, which can take a long time depending on network speed. In a separate session, you can SSH as the ksadmin user and run the ls command in the /home/ksadmin directory to check the progress of the upgrade file upload.

   3. If the cmupdate prepare command output identifies any problems, correct them and repeat the cmupdate prepare command.

      These may include:

      • Insufficient disk space.

        • Check the retention setting for scheduled backups on each node to see if many backup files are being retained. You might need to manually transfer some backup files off the appliance.

        • Disable local database audit logging on one node. Local database logging can occupy a lot of disk space.

      • Cluster nodes not in active state.

        This message indicates that one or more nodes are not fully joined to the cluster, with a status of ready. Check the status of cluster nodes. If the status is joining, wait for the join operation to complete. If the status is nodes are in different states across databases, not clustered, killed, down, or unknown, attempt a rejoin operation.

      • Audit records are being written to the database.

        Disable local database audit logging on one node.

      • Cannot verify the signature of the upgrade file.

        Re-download the upgrade file from support portal. Contact customer support if the issue happens again.

      • Cluster nodes have different versions.

        Use one of the online in-place cluster upgrade or cluster remove/rebuild methods to bring up all the nodes to the same starting CipherTrust Manager version.

5. Stop as much CipherTrust Manager traffic as possible. The cluster will not be available for part of the ongoing upgrade.

6. Run the cmupdate upgrade command, which checks the uploaded upgrade file matches the specified upgrade file, disables the REST API for all nodes, and then upgrades each node and re-enables the REST API. During some of the upgrade, the cluster is unavailable to incoming traffic. You require the path to the upgrade file, the SSH keys of each CipherTrust Manager node, and the IP address or hostname for at least one node.

   1. Run the command.

      cmupdate upgrade --identities <ssh_private_key_1,ssh_private_key_2 ...> --nodes <hostname_or_IP_address_of_a_node> --file <upgrade_file_path> --all
      

   2. You are asked to confirm the hostnames or IP addresses of every node in the cluster.

      The following hosts are part of the cluster and will be upgraded: 1.1.1.1,1.1.1.2 Continue (y/n)? y
      

7. Once the upgrade completes and the REST API is re-enabled, you can restart any client requests.

Upgrade Using the Cluster Remove/Rebuild Method

Supported upgrade paths are the same as for system upgrade of an unclustered appliance.

Prerequisite for Cluster Remove/Rebuild Method

Plan for downtime when clients cannot make requests to the cluster and manually block client access to the CipherTrust Manager IP addresses/hostnames in your network before starting.

To Perform an Upgrade Using Cluster Remove/Rebuild Method

1. On one of the cluster nodes, create and download a backup with corresponding backup key, in case there are any problems.

2. Remove all nodes from the cluster except one.

3. Delete the cluster configuration on the remaining node.

   $ ksctl cluster delete
   

4. Perform the upgrade on that remaining node.

   1. Transfer the archive file to the CipherTrust Manager. You require the private SSH key associated with the ksadmin account. The command and supported private key format depends on the operating system you are transferring the archive file from.

      Source Operating System	Example Command Syntax	Required Private Key Format
      Linux	scp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	OpenSSH format
      Windows	pscp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	PuTTY PPK format

   2. SSH into the CipherTrust Manager as ksadmin and ensure there sufficient disk space available. The required disk space is 5 times the size of the upgrade file. For example, if the upgrade file is 5 GB, 25 GB of free space is required. Use df -h / to view available space.

   3. Run the following command to upgrade the device.

      $ sudo /opt/keysecure/ks_upgrade.sh -f <archive_file_path>
      

      Here, <archive_file_path> specifies the CipherTrust Manager path to the signed archive file.

      The signature of the archive file is verified and the upgrade is applied.

   4. Reboot the appliance as prompted.

   5. From the ksadmin session, run systemctl status keysecure to ensure that the CipherTrust Manager services have started. Alternatively, you can visit the CipherTrust Manager web console or attempt to connect with the ksctl CLI.

   6. If you need to verify the new version, look for the version displayed in CipherTrust Manager banner in the SSH session on reboot, as shown in the screenshot. Alternatively, log in the web console UI and click the information icon at the top right to display version information.

      

5. After the appliance reboots, re-build the cluster by creating a new cluster on this node.

6. Perform the upgrade on all other removed nodes, as outlined in step 4.

   Note

   If a previously used node is to be re-used, the cluster must first be deleted from that system.

7. Join new instances to the cluster.

8. Re-establish client access to the cluster.

9. Re-download and re-install ksctl to access commands and settings introduced in the new CipherTrust Manager software version.

   Note

   Previous versions of ksctl still work with new CipherTrust Manager software for existing features and settings.

Upgrade Troubleshooting

The following section lists some commonly encountered errors during upgrade and their solutions.

Occasional Error for Devices Upgraded from NextGen KeySecure 1.6 or 1.7

CipherTrust Manager appliances that at one time ran NextGen KeySecure software versions 1.6 or 1.7 occasionally fail to upgrade on future versions. The error reported is E: Unable to locate package docker-compose-plugin.

Solution: Contact customer support to resolve this state.