TDPaaS Deployment

Thales Data Platform as a Service (TDPaaS), also referred to as Data Management Service, is a SaaS component and an alternative to the Hadoop Services offered by on-prem Thales Data Platform (TDP). TDPaaS is server-less and eliminates the need for manual administration and management of the services.

Prerequisites

• Ensure that your instance can connect to following URLs over the standard HTTPS port 443

  Region	URL
  us-central1	https://us.tdpaas.dpondemand.io
  southamerica-east1	https://tdpaas.dpondemand.io/, https://latam.tdpaas.dpondemand.io
  europe-west3	https://eu.tdpaas.dpondemand.io

• (Recommended) Whitelist the URL to facilitate communication with TDPaaS without issues.

Regions

TDPaaS is a cloud-based service for which you must choose a region to save the scan and report data. It is advised to select the region based on the proximity and sovereignty requirements.

Currently, TDPaaS supports the following regions:

• us-central1

• southamerica-east1

• europe-west3

Configuring DDC for TDPaaS

For configuring TDPaaS, you don't need to add any external connections or configure any settings.

To configure your CipherTrust Manager instance with TDPaaS:

1. Log in to CipherTrust Manager.

2. Go to Data Discovery and Classification > Settings > Cloud Management.

3. Select the Data Management tab.

4. Select a region from the Region dropdown.

   Note

   Region cannot be modified once the TDPaaS is configured. It is advised to choose the region based on the proximity and sovereignty requirements.

5. Click Get Credentials.

   The credentials are generated and saved with the CipherTrust Manager to be used by Data Discovery and Classification. A unique Customer ID is created corresponding to the current CipherTrust Manager instance.

6. Click the Download Credentials button to download and save the credentials for future use.

   Caution

   It is recommended to store the TDPaaS credentials in a secure location.

   These credentials are essential for re-establishing the TDPaaS connection if settings are corrupted or lost. To restore the connection, you must upload the downloaded credentials when prompted by CipherTrust Manager. Without the credentials, restoring TDPaaS connection is not possible.

For troubleshooting errors when working with TDPaaS, refer to Troubleshooting section.

Migrating data from on-premises TDP to TDPaaS

You can migrate on-premises TDP data to TDPaaS without any loss. The migration transfers all the data related to scans, reports, and logs to the cloud (TDPaaS). During migration, all DDC functions that are dependent on TDP (eg. scan runs and report generation) are blocked and you cannot perform any read-write operations to interact with on-premises TDP or TDPaaS. After the migration is complete, scans and other tasks can be performed on the migrated data as usual.

Prerequisite

• Ensure no scans are in running state in root domain and any subdomain.

• Ensure report generation is not in progress in root domain and any subdomain.

• Ensure on-premises TDP is not in use.

• Obtain the zip file containing the necessary files to start the migration process. Contact Thales Customer Support for the zip file.

Migrate data to TDPaaS

1. Configure CipherTrust Manager and Data Discovery and Classification application.

   a. Log in to CipherTrust Manager as a root user.

   b. Go to the API playground and set the value parameter of the update API /v1/ddc/system-settings/properties/DDC_TDP_DATA_MIGRATE to true.

   This action will block DDC functions that are dependent on TDP, such as scan runs and report generation. If a scan is running or report generation is in progress, this API will display an error.

   c. Provision TDPaaS.

2. Download on-premises TDP and TDPaaS credentials in a JSON file.

   TDPaaS

   1. Go to the API playground and set the value parameter of the API /v1/ddc/system-settings/properties/FF_DDC_UNHIDE_DOWNLOAD_CREDENTIALS to true.

   2. Go to Settings > Cloud Management > Data Management.

   3. Click Download Credentials to save the credentials in a JSON file.

   On-premises TDP

   1. In the CipherTrust Manager console, open the API playground.

   2. Run the GET method for the /v1/connectionmgmt/services/hadoop/connections API.

   3. Save the response in a JSON file.

3. Use the CLI utility to migrate data.

   a. Extract the content of the ZIP file on the host machine that can access on-premises TDP and TDPaaS without restrictions.

   b. Execute the tdp-data-migrate-<version> binary to start the CLI utility.

   Note

   The binary file is executable only on Linux machines.

   c. Use the tdp-data-migrate command to provide CipherTrust Manager, on-premises TDP, and TDPaaS credentials. See Command reference for tdp-data-migrate.

   d. Execute the ./tdp-data-migrate start command with root user privileges.

   This will start the migration process. If migration stops or fails for any reason, you can restart the process by executing the binary again. The migration will resume from the point at which it was previously stopped.

   Note

   The command execution creates credentials and log directories. Do not alter or modify any details within these directories.

4. Post migration.

   a. Set the value parameter of the update API /v1/ddc/system-settings/properties/DDC_TDP_DATA_MIGRATE to false. DDC will become operable and the system will start using TDPaaS for data management.

   b. Set the value parameter of the API /v1/ddc/system-settings/properties/FF_DDC_UNHIDE_DOWNLOAD_CREDENTIALS to false.

   c. Access all scans and reports on the CipherTrust Manager (that were run using TDP) as usual.

Revert to on-premises TDP

To revert to on-premises TDP:

1. Set the value paramter of the update API /v1/ddc/system-settings/properties/DDC_TDP_DATA_MIGRATE to true.

2. In CipherTrust Manager, go to Data Discovery and Classification > Settings > Hadoop Services.

3. Configure the HDFS and Livy services.

Command reference for tdp-data-migrate

The tdp-data-migrate command simplifies and automates the process of migrating data from on-premises TDP to TDPaaS. It accepts credentials for on-premises TDP and TDPaaS to perform the migration process. You can provide the credentials either using [flags] or [commands].

Syntax:

    tdp-data-migrate [flags]
    tdp-data-migrate [command]

Commands:

Flags: