Adding, Removing, Replacing, or Reconnecting HA Group Members

Home >	Administration Guide > High-Availability (HA) Configuration and Operation > Adding, Removing, Replacing, or Reconnecting HA Group Members

Adding, Removing, Replacing, or Reconnecting HA Group Members

This section describes how add a new member to an HA group, reconnect an offline member, or replace a failed unit.

Adding or Removing an HA Group Member

Use the following LunaCM commands to add or remove a normal or standby member to or from an HA group:

•hagroup addmember

•hagroup addstandby

•hagroup removemember

•hagroup removestandby

See hagroup in the LunaCM Command Reference Guide for detailed descriptions and syntax for each hagroup command.

Note: You must restart the application to have the added or removed member recognized.

Reconnecting an Off-line Unit

In HA mode, if an HSM appliance goes off-line/drops-out (due to failure, maintenance, or other reason), the application load is spread over the remaining HSM Partitions on appliances in the HA Group. When the unit is restarted, the application does not need be stopped and restarted, before the re-introduced unit can be used by the application. For the unit that was withdrawn (or for a replacement unit), if it was powered off for more than a short outage, you must re-activate the Partitions before they can be re-included into the HA Group.

The following two reconnection scenarios are available:

To recover the same group member

1.Restart the failed member and verify that it has started properly.

2.Do not perform a manual re-synchronization between the members. Instead, use the following LunaCM command:

lunacm:> ha -recover -group <group_name>

To replace a failed group member with a new appliance

1.Configure the new SafeNet Network HSM as follows:

–name it differently from the failed member appliance, The name must be different to avoid any possibility of conflict between the old and new SSL certificates, which incorporate the hostnames of the respective appliances.

–make it part of the same cloning domain as others in the HA group. At initialization, the HSM gets its cloning domain from the same red domain PED Key.

If you require that the replacement appliance must have the same name as the replaced appliance, then you will need to stop your application before introducing the new appliance.

2.Create a partition with the same characteristics as others in the HA group ( password, autoActivation, auto MofN, client assignments, etc.).

3.Do not delete the failed SafeNet Network HSM member from the Chrystoki.conf (Unix/Linux) or Crystoki.ini (Windows) configuration file.

4.Determine the serial number of the failed member partition.

5.Retrieve the server certificate of the new SafeNet Network HSM.

6.Replace the failed SafeNet Network HSM with the new one using the following VTL command:

vtl replaceServer -o <oldServerName> -n <newServerName> -c <newServerCertFile>

7.Add the new partition of the new SafeNet Network HSM to the HA group using the relevant command below:

–lunacm:> ha -addMember -group <group number> -serialNum <serialnumber> -password <password>

–lunacm:> ha -addMember -group <group number> -slot <slotnumber> -password <password>

8.Remove the failed member from the HA group, using the relevant command below:

–lunacm:> ha -removeMember -group <groupNumber> -serialNum <serialnumber>

– lunacm:> ha -removeMember -group <groupNumber> -slot <slotnumber>

9.Do not perform a manual re-synchronization between the members. Instead, use the following command:

–lunacm:> ha -recover -group <group_name>

Replacing a Failed SafeNet Network HSM

Before getting into replacing HSMs in an HA group, this first section describes relevant system conditions and settings to have a SafeNet Network HSM configured and in an authenticated relationship with a client computer. In particular, we are interested in the client-side config file and the client's certificate folder in ordinary, single-appliance mode, and then in HA. You would already have set up the a SafeNet Network HSM as described in the configuration manual, for network setup and creation of the appliance-side certificate (see Generate a New HSM Server Certificate).

Chrystoki.ini before client-side certificate creation

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\ClientNameCert.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\ClientNameKey.pem

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1

1.Create client-side certs (see vtl createCert in the Utilities Reference Guide).

Generated client certificates

Chrystoki.ini after client-side certificate creation

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1

2.Copy SafeNet Network HSM server.pem to client.

Note: At this point there are still no certificates in cert\server directory.

3.Use “vtl addserver” to register the SafeNet Network HSM with the client.

CAFile.pem is generated in the cert\server directory.

Cert\server directory after CAFile.pem is generated

Crystoki.ini after "vtl addserver"

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem 
ServerName00=20.1.1.20
ServerPort00=1792

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1

vtl verify results

C:\Program Files\SafeNet\LunaClient>vtl verify


The following SafeNet Network HSM Slots/Partitions were found:

Slot	Serial # 	Label
====	========	=====
1	154702010	p1

C:\Program Files\SafeNet\LunaClient>

Replace a SafeNet Network HSM Using the same IP

For an existing HA group, bring in a replacement SafeNet Network HSM.

1.Change the IP of the new appliance to match the one that was removed.

2.Perform RegenCert on the new SafeNet Network HSM.

Note: “vtl verify” on client at this time would fail because the cert that the client has is for the old, removed SafeNet Network HSM.

3.Execute “vtl deleteserver –n <original IP>

Deleting old SafeNet Network HSM from Client

C:\Program Files\SafeNet\LunaClient>vtl listservers
Server: 20.1.1.20

C:\Program Files\SafeNet\LunaClient>vtl deleteserver -n 20.1.1.20
Server: 20.1.1.20 successfully removed from server list.

C:\Program Files\SafeNet\LunaClient>

Contents of cert\server after “deleteserver” (CAFile.pem has been deleted)

4.Copy new server.pem to client

Copying new server.pem to client

C:\Program Files\SafeNet\LunaClient>pscp admin@20.1.1.20:server.pem . 
admin@20.1.1.20's password:  
server.pem		| 1 kB | 1.1 kB/s | ETA: 00:00:00 | 100%

5.Run vtl addserver using new server.pem

vtl addserver using new server.pem

C:\Program Files\SafeNet\LunaClient>vtl addserver -n 20.1.1.20 -c server.pem
New server: 20.1.1.20 successfully added to server list.

C:\Program Files\SafeNet\LunaClient>

6.Run vtl verify.

vtl verify results

C:\Program Files\SafeNet\LunaClient>vtl verify


The following SafeNet Network HSM Slots/Partitions were found:

Slot	Serial # 	Label
====	========	=====
1	154702010	p1

C:\Program Files\SafeNet\LunaClient>

Summary

If a SafeNet Network HSM must be replaced, the old IP can be used, but the SafeNet Network HSM certificate must be regenerated. The IP must be removed from the server list on the client and then added back using the new “server.pem”

Client side requirements review:

•Use vtl deleteserver to remove IP from list and delete CAFile.pem from cert\server

•Copy “new” server.pem to client

•Use vtl addserver to re-add IP and create CAFile.pem

Client-side - Reconfigure HA If a SafeNet Network HSM Must Be Replaced

1.Note HA partition serial numbers

C:\Program Files\SafeNet\LunaClient>vtl verify
The following SafeNet Network HSM Slots/Partitions were found:
Slot	Serial # 	Label
====	========	=====
1	154702011	HA1
1	154702012	HA2     

C:\Program Files\SafeNet\LunaClient>

2.Run "lunacm ha -newGroup..."

A group is created with HA1 as Primary.

C:\Program Files\SafeNet\LunaClient>vtl haadmin -newGroup -label SomeHAGrp -serial 154702011 -password userpin
New  group with label "SomeHAGrp" created at group number 1154702011. 
Group configuration is:

		 HA Group label:  SomeHAGrp
		HA Group Number:  1154702011
		HA Group Slot #:  unknown
		Synchronization:  enabled
		  Group Members:  154702011
		Standby Members:  <none>
			In Sync:  yes

C:\Program Files\SafeNet\LunaClient>

Crystoki.ini after HA group is created

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem 
ServerName00=20.1.1.20
ServerPort00=1792

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1   

[VirtualToken]   
VirtualToken00Label=SomeHAGrp  
VirtualToken00SN=1154702011   
VirtualToken00Members=154702011 

[HASynchronize]   
SomeHAGrp=1

3.Add a secondary SafeNet Network HSM partition to the HA group with lunacm:> ha - addMember.

lunacm:> ha -addMember -group SomeHAGrp -serialNum 154702012 -password userpin
New  group with label "SomeHAGrp" created at group number 1154702011. 
Group configuration is:

		 HA Group label:  SomeHAGrp
		HA Group Number:  1154702011
		HA Group Slot #:  6
		Synchronization:  enabled
		  Group Members:  154702011, 154702012  
		Standby Members:  <none>
			In Sync:  yes

Please use the command 'vtl haAdmin -synchronize' when
you are ready to replicate data among all members of the
HA grou. (If you have additional members to add, you might
wish to wait until you have added them before synchronizing
to save time by avoiding multiple synchronizations.)  

C:\Program Files\SafeNet\LunaClient>

Crystoki.ini after second HA member is added

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem 
ServerName00=20.1.1.20
ServerPort00=1792

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1   

[VirtualToken]   
VirtualToken00Label=SomeHAGrp  
VirtualToken00SN=1154702011   
VirtualToken00Members=154702011, 154702012  

[HASynchronize]   
SomeHAGrp=1

Crystoki.ini after HA Only is enabled

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem 
ServerName00=20.1.1.20
ServerPort00=1792

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1   

[VirtualToken]   
VirtualToken00Label=SomeHAGrp  
VirtualToken00SN=1154702011   
VirtualToken00Members=154702011, 154702012  

[HASynchronize]   
SomeHAGrp=1   

[HAConfiguration]   
HAOnly=1

Crystoki.ini after "autorecovery" is enabled

[Chrystoki2]
LibNT=C:\Program Files\SafeNet\LunaClient\cryptoki.dll

[LunaSA Client]
SSLConfigFile=C:\Program Files\SafeNet\LunaClient\openssl.cnf
ReceiveTimeout=20000
NetClient=1
ServerCAFile=C:\Program Files\SafeNet\LunaClient\cert\server\CAFile.pem
ClientCertFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem
ClientPrivKeyFile=C:\Program Files\SafeNet\LunaClient\cert\client\20.1.1.20.pem 
ServerName00=20.1.1.20
ServerPort00=1792

[Luna]
DefaultTimeOut=500000
PEDTimeout1=100000
PEDTimeout2=200000
PEDTimeout3=10000

[CardReader]
RemoteCommand=1   

[VirtualToken]   
VirtualToken00Label=SomeHAGrp  
VirtualToken00SN=1154702011   
VirtualToken00Members=154702011, 154702012  

[HASynchronize]   
SomeHAGrp=1   

[HAConfiguration]   
HAOnly=1 
reconnAtt=500

4.Show HA configuration results with vtl haAdmin -show

C:\Program Files\SafeNet\LunaClient>vtl haadmin -show 


================== HA Global Configuration Settings ============  
              HA Auto Recovery:  enabled       
   Maximum Auto Recovery Retry:  500  
   Auto Recovery Poll Interval:  60 seconds   
                    HA Logging:  disabled   
            Only Show HA Slots:  yes


================== HA Group and Member Information ============

		 HA Group label:  SomeHAGrp
		HA Group Number:  1154702011
		HA Group Slot #:  1
		Synchronization:  enabled
		  Group Members:  154702011, 154702012  
		Standby Members:  <none>
			

Slot #  Member S/N 		Member Label	Status
======	 ========	  	============	======
   -	 154702011		   	HA1	alive   
   -	 154702012		  	HA2	alive    

C:\Program Files\SafeNet\LunaClient>  >

Replacing the Secondary HA Group Member

When the SafeNet Network HSM to be replaced, in an HA Group, is a secondary member, the process is similar to above.You must delete the secondary from the HA Group and re-add it with the new partition serial number. It is not necessary to delete and recreate the group.

If a SafeNet Network HSM must be replaced, the old IP address can be used, but the SafeNet Network HSM certificate must be regenerated. The IP address must be removed from the server list on the client and then added back using the new “server.pem” received from the replacement SafeNet Network HSM.

If the SafeNet Network HSM being replaced is the Primary, you must delete the HA Group and recreate it using the new Primary SafeNet Network HSM partition serial number and then add the original Secondary SafeNet Network HSM partition serial number - the cert from the original Secondary is already in place on the client, and no change is needed to that.