Replication of IDT Metadata Files Across Members of a Clique

For Teradata, metadata for IDT-Capable GuardPoints is stored in external files in the CTE metadata directory. Because the Teradata appliance is a cluster of multiple hosts/clients that share access to the same devices across multiple nodes, a metadata file must be replicated across the nodes that are in the same clique within the cluster, including Hot Standby nodes. Replication is required when the initial data transformation has been completed, and it is required again during subsequent rekeys. The availability of the metadata files on all members of the clique is critical for high availability of Teradata database in the event of a node failure because the data on the device cannot be accessed without the encryption key stored in the metadata.

Upon completion of data transformation on each device, the CTE Agent on that device automatically replicates the metadata file for that device to the other nodes in the clique using the Teradata pcl command. CTE uses pcl to both determine the other members of the clique and to replicate the metadata to those other members. When the metadata is replicated on the remote nodes, any existing metadata for the recently-transformed device already on those nodes is replaced with the updated metadata files sent by the CTE Agent. This replacement is achieved by sending the updated metadata file to all of the remote nodes through pcl and replacing each existing metadata file on the remote nodes with the most recent metadata files.

Specific Issues to Consider

This section describes specific problems that may be encountered during data transformation and metadata replication. Manual user intervention will be required to recover from the reported issues. For troubleshooting and recovery steps, see Alerts and Errors.

General PCL Error

If the pcl command fails during the replication process, a message indicating the error will be logged. The messages are tagged with IDT-TD-ALERT. For example:

IDT-TD-ALERT: Node did not respond to pcl command
IDT-TD-ALERT: Failed to distribute IDT-Capable metadata file to remote nodes

These errors indicate that the node specified in the first error did not respond to a pcl command during metadata distribution. As a result, the metadata distribution must be manually performed across the clique before access to the device is possible.

Offline Node in Clique During Data Transformation

If a node is offline during data transformation and metadata replication process, CTE will log a message that metadata replication to the target node failed. The administrator will be required to manually replicate the metadata file to the offline node when the node comes online. The metadata file must be replicated before the database is brought up.

To do so:

Run the voradmin td distribute <device name> command to distribute the metadata file of each device. The command will copy, or update, the metadata file on the agent that has come online. For example:
```
 voradmin td distribute \
/dev/disk/by-id/tdmp-360080e500043092c0000b46f5c34c018-part3
```

Adding a New Node to a Clique

When you add a new node to an existing clique, the metadata files of all disks shared in the clique must be manually replicated to the newly added node before any of the guarded devices on the new node are enabled. This should be done manually using the voradmin td distribute command, as described above, by the Teradata administrator as part of joining the cluster.

Interoperability with Host Groups

For Teradata, when you create a new IDT-Capable GuardPoint using the voradmin idt config xform command, or when you subsequently rekey an existing IDT-Capable GuardPoint with the voradmin idt rekey command, the device must be intialized and guarded on one and only one of the nodes in the cluster. That means the IDT-Capable GuardPoint cannot be part of a client group when an IDT-Capable GuardPoint is created or rekeyed because membership in a client group means that any data transformation on any member of the client group is initiated for that member on all nodes in the client simultaneously. In the case of a Teradata cluster, multiple nodes simultaneously trying to perform data transformation on a particular device can lead to data corruption of all data on the entire device.

After the device has been guarded, or rekeyed, and the metadata files have been replicated to other members of the Teradata clique, then you can then rejoin the clientwith the client group.

To configure a new device whose host/client is part of a host/client group:

Make sure that there is no GuardPoint for the device at the client group level.
Designate one of the nodes in the cluster as the node that you will use to initialize the GuardPoint and for the initial data transformation when the device is guarded for the first time.
On the designated node, initialize the device using the voradmin idt config -external xform <device name> command. For details, see Initialize and Guard the Database Devices Using the Standard Initialization Method.
Guard the device on the designated node using an In-Place Data Transformation policy. For details, see Guard the Devices as IDT-Capable GuardPoints on CM.
Wait for the data transformation to complete on the host and for CTE to replicate the metadata to the other members of the clique.
You can verify that the metadata file has been distributed to the other nodes in the clique by running md5sum /var/opt/teradata/vormetric/vte-metadata-dir/vormetric/secvm_<device>_metadata on each node in the clique.
Remove the IDT-Capable GuardPoint you added earlier in this procedure.
Guard the device through the client group to make sure that all nodes in the cluster recognize it as guarded.

To rekey a new device whose host/client is part of a host/client group:

Unguard the IDT-Capable GuardPoint through the client group and make sure that it has been removed from all nodes in the cluster.
Designate one of the nodes in the cluster as the node that you will use to prepare the GuardPoint for rekeying and to perform the data transformation when the device is guarded with the new policy.
On the designated node, prepare the device for rekey using the voradmin idt rekey <device name> command.
On the other nodes in the clique, make sure that the device metadata has been renamed running ls /var/opt/teradata/vormetric/vte-metadata-dir/vormetric/secvm_<device>_metadata* on each of the other nodes. On these other nodes, the metadata file for the device should be renamed to /var/opt/teradata/vormetric/vte-metadata-dir/vormetric/secvm_<device>_metadata_xforming.
Guard the device on the designated node with the In-Place Data Transformation policy that specifies the new key you want to use for the device.
Wait for the data transformation process to complete, and then make sure that the metadata for the device has been updated on the other nodes in the clique and the renamed metadata files have been removed. Each node should have identical copies of /var/opt/teradata/vormetric/vte-metadata-dir/vormetric/secvm_<device>_metadata and /var/opt/teradata/vormetric/vte-metadata-dir/vormetric/secvm_<device>_metadata_xforming should not exist on any node.
Remove the IDT-Capable GuardPoint that you created earlier in this procedure on the designated node.
Guard the device through the client group to make sure that all nodes in the cluster recognize it as guarded.

Suggest A Change