Tokenize and detokenize large data set
Bulk migration utility is a command line tool provided with CADP for Java. It enables tokenization and detokenization of a large data set. The utility is controlled through variables exposed in the configuration files placed at the CADP_for_JAVA/lib/ext directory. The following operations can be performed using the utility:
File-to-File Tokenization
File-to–File Detokenization
Caution
While performing bulk tokenization and detokenization, the utility cannot identify and skip erroneous records. Files with an error are rejected. Make sure the input data files provided during tokenization and detokenization adheres to the criteria mentioned in the properties files.
Associated files
CADP for Java performs bulk tokenization and detokenization operations based on the configurations set in the migration.properties and detokenization.properties files. These files are located in the CADP_for_JAVA/lib/ext directory.
The
migration.propertiesfile is used for file-to-file tokenization of bulk data involving both plaintext and encrypted texts.The
detokenization.propertiesis used for file-to-file detokenization of tokens.
These properties are largely self-documenting. Open these files to become familiar with the capabilities and usage of bulk migration utility.
Supported operations
The following table outlines the operations that can be performed by bulk migration utility:
| Operation | Input Data | Format of Input Data | Properties File Used |
|---|---|---|---|
| Tokenization (File-to-File) | Plaintext value | — Delimited — Positional | migration.properties |
| Tokenization (File-to-File) | Plaintext value (stream data) | Positional | migration.properties |
| Detokenization (File-to-File) | Tokens | — Positional — Delimited | detokenization.properties |
| Detokenization (File-to-File) | Tokens (stream data) | Positional | detokenization.properties |
If the input value for bulk tokenization is already in encrypted form, the utility first decrypts the data and then tokenizes it as per the parameters set in the migration.properties file.
Advantages of bulk migration
The bulk migration utility uses multithreaded environment to provide high performance. The following parameters can be modified in the migration.properties and detokenization.properties files as required for high performance.
| Parameters in migration.properties File | Parameters in detokenization.properties File |
|---|---|
| Threads.BatchSize | Threads.BatchSize |
| Threads.CryptoThreads | Threads.FormatterThreads |
| Threads.TokenThreads | Threads.DetokenThreads |
| Threads.PollTimeout | Threads.PollTimeout |
How to run bulk migration
The utility doesn't require separate installation. Run the following commands to perform bulk tokenization and detokenization:
Bulk tokenization
Use any of these commands:
Bulk detokenization
Use any of these commands:
Usage
The following tables describes the available options provided in the utility:
| Usage | Description |
|---|---|
| -t | Used for tokenization. |
| -d | Used for detokenization |
| KSU | Key Manager user name. |
| KSP | Password associated with the Key Manager user name. |
| Key | Key for tokenization or detokenization. |
