Secure Messaging

An optional trusted channel called the Secure Message System (SMS) may be enabled. It is disabled by default. This system enables applications to securely communicate with HSMs over the PCI bus interface, or across a network.

A trusted channel is created on-demand by the operator but may be terminated by either the HSM or the operator. Either the HSM or application may be configured to require a trusted channel to be created before cryptographically sensitive services can be provided. For the HSM to be compliant to FIPS 140-2 Level 3 operation the HSM must be configured in this way. However it is also possible for the application to request and use a trusted channel even though the HSM is not configured to require them.

The HSM can manage multiple simultaneous trusted channels, each of which will have its own set of randomly generated session keys for message encryption/decryption and message signing/verification. The negotiation of these session keys is based on a shared secret known by both the application and the HSM.

ProtectServer uses Anonymous Diffie-Hellman (ADH) secure messaging. The shared secret is a triple-length DES key derived from an Anonymous Diffie-Hellman key.

To configure and enable SMS

1.Configure secure messaging mode.

You may need to change the session key rollover default configuration. See the section Configuring Session Protection for further information.

2.Configure session protection and enable SMS.

The SMS is enabled by setting one or more security flags that control how the SMS functions. By default these flags are cleared so SMS is disabled. To enable and configure SMS, see the section Configuring Session Protection.

Messaging Mode Configuration

Anonymous Diffie-Hellman (ADH) mode selection

With the No Clear PINs flag set (see No Clear PINs), the ProtectServer Client software uses the default, NIST-approved ADH2 mode for secure messaging using SHA-512-based MAC. The default mode can be overridden and set to the legacy ADH mode (which uses SHA-1).

To change the SMS mode to ADH, set the configuration item ET_PTKC_SMS_MODE to ADH. For more information about this configuration item, refer to Secure Messaging Configuration Items.

$ ET_PTKC_SMS_MODE=ADH

Configuring Session Key Rollover

Session key rollover involves dynamically changing the keys used to perform encryption/decryption between the application and the hardware security module (HSM).

Two mechanisms can be used to trigger session key rollover.

1.The first mechanism triggers session key rollover once a preset number of blocks have been encrypted or decrypted by the application.

2.The second mechanism triggers session key rollover after a preset number of hours have elapsed since a connection was established with the HSM.

Each of these mechanisms is covered in more detail in the following sections.

Preset Number of Blocks Trigger

This mechanism is used to trigger session key rollover once a preset number of blocks have been encrypted or decrypted by the application. The default value for the number of blocks is 2³². This default value can be overridden by setting the configuration item ET_PTKC_SMS_BLOCKS to the desired value. For more information about this configuration item, refer to Secure Messaging Configuration Items.

For example, on a UNIX machine, to temporarily change the key rollover trigger so that key rollover occurs after 10,000 blocks have been encrypted or decrypted the following shell commands would be used:

$ ET_PTKC_SMS_BLOCKS=10000
$ export ET_PTKC_SMS_BLOCKS

This change can be made at the temporary, user or system levels on both UNIX and Windows platforms. Refer to Configuration Items for further details on how to go about this if required.

Preset Number of Hours Trigger

This mechanism is used to trigger session key rollover after a preset number of hours have elapsed since a connection was established with the HSM. The default value for the number of hours is 24. This default value can be overridden by setting the configuration item ET_PTKC_SMS_HOURS to the desired value. For more information about this configuration item, refer to Secure Messaging Configuration Items.

For example, on a UNIX machine, to temporarily change the key rollover trigger to occur after 4 hours have elapsed, the following shell commands would be used:

$ ET_PTKC_SMS_HOURS=4
$ export ET_PTKC_SMS_HOURS

This change can be made at the temporary, user or system levels on both UNIX and Windows platforms. Refer to Configuration Itemsfor further details on how to go about this if required.

Configuring Session Protection

When applications establish a session with a hardware security module (HSM) using ProtectToolkit-C, secure messaging layer activation depends upon:

>Security flag settings (the security policy) stored in tamperable memory inside the HSM by the administrator

>Any additional security flag settings specified by users where they wish to increase the level of security used. These user specified security flag settings are stored in the Secure Messaging Policy Register (SMPR) on the client machine.

Generally, the HSM-stored security flag settings are sufficient so the Secure Messaging Policy Register is rarely used.

NOTE   Session protection is only applied to Cryptoki functions that use a session handle returned from a previous call to C_OpenSession().

HSM Stored Security Flags

HSM stored security flags can be set at the local machine regardless of whether the HSM is located in the same machine as the application (PCI mode) or remotely (network mode). In the latter case it will be necessary to know the administrator’s password for the server machine as this must be entered before any server side changes can be made.

The following table lists those flags that, when set for HSM storage, effect secure messaging. For further information about these flags please see Security Policies and User Roles.

Flag	Secure Messaging Effect
No clear PINs	Only messages sent to the HSM that contain sensitive data are encrypted
Auth Protection	Only messages sent to the HSM are signed
Full Secure Message Encryption	All messages sent to and from the HSM are encrypted
Full Secure Message Signing	All messages sent to and from the HSM are signed

To Set HSM-Stored Security Flags

These flags can be set using the ProtectToolkit-C ctconf utility command, ctconf -f<flags>. Refer to Security Policies and User Roles for full details on security policies, setting flags and the use of this command.

SMPR Security Flags

The Secure Messaging Policy Register (SMPR) flag settings augment the HSM settings discussed above and are stored on the client machine by assigning configuration item values.

As the client may access more than one HSM the SMPR can store a unique set of settings for each accessible HSM if required. Each HSM is identified by its serial number for SMPR storage purposes.

To Set SMPR Security Flags

1.Obtain the serial number of the HSM.

This can be done by executing the command ctconf -a<device> from a command line, where <device> is the number of the HSM in the list of HSMs.

2.Create the following configuration item:

ET_PTKC_<serial>_SMPR

...where <serial> is the serial number of the HSM found in step 1.

This change can be made at the temporary, user or system levels on both UNIX and Windows platforms. Refer to Configuration Items for further details on how to go about this if required.

3.Set one or more flags by assigning a value to the configuration item. For more information about the valid values for this configuration item, refer to Secure Messaging Configuration Items. For example, if both Auth Protection and Auth Replies are required, assign the value SR.