Secure Messaging
An optional trusted channel called the Secure Message System (SMS) may be enabled. It is disabled by default. This system enables applications to securely communicate with HSMs over the PCI bus interface, or across a network.
A trusted channel is created on-demand by the operator but may be terminated by either the HSM or the operator. Either the HSM or application may be configured to require a trusted channel to be created before cryptographically sensitive services can be provided. For the HSM to be compliant to FIPS 140-2 Level 3 operation the HSM must be configured in this way. However it is also possible for the application to request and use a trusted channel even though the HSM is not configured to require them.
The HSM can manage multiple simultaneous trusted channels, each of which will have its own set of randomly generated session keys for message encryption/decryption and message signing/verification. The negotiation of these session keys is based on a shared secret known by both the application and the HSM.
ProtectServer uses Anonymous Diffie-Hellman (ADH) secure messaging. The shared secret is a triple-length DES key derived from an Anonymous Diffie-Hellman key.
To configure and enable SMS
1.Configure secure messaging mode.
You may need to change the session key rollover default configuration. See the section Configuring Session Protection for further information.
2.Configure session protection and enable SMS.
The SMS is enabled by setting one or more security flags that control how the SMS functions. By default these flags are cleared so SMS is disabled. To enable and configure SMS, see the section Configuring Session Protection.
Messaging Mode Configuration
Anonymous Diffie-Hellman (ADH) mode selection
With the No Clear PINs flag set (see No Clear PINs), the ProtectServer Client software uses the default, NIST-approved ADH2 mode for secure messaging using SHA-512-based MAC. The default mode can be overridden and set to the legacy ADH mode (which uses SHA-1).
To change the SMS mode to ADH, set the configuration item ET_PTKC_SMS_MODE to ADH.
$ ET_PTKC_SMS_MODE=ADH
NOTE PTK firmware versions 5.01.00 and newer support ADH2 only. ADH is included for use with legacy firmware older than 5.01.00. Setting the SMS mode to ADH with newer firmware will return an error message.
Configuring Session Key Rollover
Session key rollover involves dynamically changing the keys used to perform encryption/decryption between the application and the hardware security module (HSM).
Two mechanisms can be used to trigger session key rollover.
1.The first mechanism triggers session key rollover once a preset number of blocks have been encrypted or decrypted by the application.
2.The second mechanism triggers session key rollover after a preset number of hours have elapsed since a connection was established with the HSM.
Each of these mechanisms is covered in more detail in the following sections.
Preset Number of Blocks Trigger
This mechanism is used to trigger session key rollover once a preset number of blocks have been encrypted or decrypted by the application. The default value for the number of blocks is 232. This default value can be overridden by setting the configuration item ET_PTKC_SMS_BLOCKS to the desired value.
For example, on a UNIX machine, to temporarily change the key rollover trigger so that key rollover occurs after 10,000 blocks have been encrypted or decrypted the following shell commands would be used:
$ ET_PTKC_SMS_BLOCKS=10000
$ export ET_PTKC_SMS_BLOCKS
This change can be made at the temporary, user or system levels on both UNIX and Windows platforms. Refer to Configuration Items for further details on how to go about this if required.
Preset Number of Hours Trigger
This mechanism is used to trigger session key rollover after a preset number of hours have elapsed since a connection was established with the HSM. The default value for the number of hours is 24. This default value can be overridden by setting the configuration item ET_PTKC_SMS_HOURS to the desired value.
For example, on a UNIX machine, to temporarily change the key rollover trigger to occur after 4 hours have elapsed, the following shell commands would be used:
$ ET_PTKC_SMS_HOURS=4
$ export ET_PTKC_SMS_HOURS
This change can be made at the temporary, user or system levels on both UNIX and Windows platforms. Refer to Configuration Items for further details on how to go about this if required.
Configuring Session Protection
When applications establish a session with a hardware security module (HSM) using SafeNet ProtectToolkit-C, secure messaging layer activation depends upon:
>Security flag settings (the security policy) stored in tamperable memory inside the HSM by the administrator
>Any additional security flag settings specified by users where they wish to increase the level of security used. These user specified security flag settings are stored in the Secure Messaging Policy Register (SMPR) on the client machine.
Generally, the HSM-stored security flag settings are sufficient so the Secure Messaging Policy Register is rarely used.
NOTE Session protection is only applied to Cryptoki functions that use a session handle returned from a previous call to C_OpenSession().
HSM Stored Security Flags
HSM stored security flags can be set at the local machine regardless of whether the HSM is located in the same machine as the application (PCI mode) or remotely (network mode). In the latter case it will be necessary to know the administrator’s password for the server machine as this must be entered before any server side changes can be made.
The following table lists those flags that, when set for HSM storage, effect secure messaging. For further information about these flags please see Security Policies and User Roles.
| Flag | Secure Messaging Effect |
|---|---|
| No clear PINs | Only messages sent to the HSM that contain sensitive data are encrypted |
| Auth Protection | Only messages sent to the HSM are signed |
| Full Secure Message Encryption | All messages sent to and from the HSM are encrypted |
| Full Secure Message Signing | All messages sent to and from the HSM are signed |
To Set HSM-Stored Security Flags
These flags can be set using the SafeNet ProtectToolkit-C ctconf utility command, ctconf -fflags. Refer to Security Policies and User Roles for full details on security policies, setting flags and the use of this command.
SMPR Security Flags
The Secure Messaging Policy Register (SMPR) flag settings augment the HSM settings discussed above and are stored on the client machine by assigning configuration item values.
As the client may access more than one HSM the SMPR can store a unique set of settings for each accessible HSM if required. Each HSM is identified by its serial number for SMPR storage purposes.
The following table lists the SMPR security mode flags, their effect on secure messaging and the configuration item values that must be assigned in order to set them.
| Flag | Secure Messaging Effect | Configuration Item Value |
|---|---|---|
| No clear PINs | Only messages sent to the HSM that contain sensitive data are encrypted | E |
| Auth Protection | Only messages sent to the HSM are signed | S |
| Auth Replies | Only messages received from the HSM are signed | R |
To Set SMPR Security Flags
1.Obtain the serial number of the HSM.
This can be done by executing the command ctconf –a<device> from a command line, where <device> is the number of the HSM in the list of HSMs.
2.Create the following configuration item:
ET_PTKC_<serial>_SMPR
...where <serial> is the serial number of the HSM found in step 1.
This change can be made at the temporary, user or system levels on both UNIX and Windows platforms. Refer to Configuration Items for further details on how to go about this if required.
3.Set one or more flags by assigning a value to the configuration item using one or more of the Configuration Item Value letters given in the table above. For example, if both Auth Protection and Auth Replies are required, assign the value SR.
