BDT Policy File
A BDT policy file includes the following attributes\parameters:
| See Also |
|---|
| Sample BDT Policy File |
Top-level Policy Attributes
BDT Policy File Top-level Policy Attributes
| Attribute | Default Value | Mandatory/Optional | Description |
|---|---|---|---|
| id | No default | Mandatory | A unique identifier for the policy. |
| name | No default | Mandatory | A unique name for the policy. |
| source | No default | Mandatory | Defines details of data input. For databases, such details include connection parameters. For files, such details include file type (CSV_FILE or FIXED_LENGTH_FILE), delimiter, encoding, and so on. |
| destination | No default | Mandatory (if inPlaceUpdate attribute is set to false) | Defines details of data output, such as where to write transformed data. For databases, such details include connection parameters. If the policy file defines an in-place transformation of data (with inPlaceUpdate attribute set to true), the destination value is ignored if defined. For files, such details include file type (CSV_FILE or FIXED_LENGTH_FILE), delimiter, encoding, and so on. |
| tables | No default | Mandatory | List of tables to be transformed. |
| createBadRecordFile | false | Optional | True or false. Set to true to create a file with .failed extension, otherwise set to false.Note: It is recommended to delete the BadRecordsFile after being used as it contains sensitive information. |
| fpemode | ASCII | Optional | ASCII or UTF. (FPE-only) Set fpemode to define which FF3-encoding mechanism to use. ASCII supports only ASCII characters and is faster than UTF. Default is ASCII. Set to UTF when using FPE to encrypt Unicode characters. Note: ASCII encryption mode is separate from ASCII character encoding. |
| inPlaceUpdate | false | Optional | Set to true to transform data in-place within the database (that is, in-place transformation), rather than the default behavior that creates a new destination table containing the transformed data. If true, the destination, destinationTable, and createDestinationTable attributes are ignored.Note: For inPlaceUpdate transformation, Primary Key column should be available in the table. If this column is not available, below error message is displayed:“Primary key column not found. Can't perform in-place update.” |
Source and Destination Attributes
For JDBC Connections
The following are the source and destination attributes for JDBC connections:
| Attribute | Default Value | Mandatory/Optional | Description |
|---|---|---|---|
| @type | No default | Mandatory | Specify the database type as one of either DB2, ORACLE, MYSQL, SQL_SERVER, HANA. |
| connectionurl | No default | Mandatory | Database connection URL. For example: jdbc:mysql://localhost:3306/world. |
| username | No default | Mandatory | Database username. |
| password | No default | Mandatory | Database password. Use bdt -e option to add encrypted password. |
| driverclass | No default | Mandatory | DBC connection driver class. For example: com.mysql.cj.jdbc.Driver. |
The following is a sample policy file excerpt for specifying a HANA database connection:
"source" : {
"@type" : "HANA",
"connectionurl" : "jdbc:sap://hostname:port",
"username" : "",
"password" : "",
"driverclass" : "com.sap.db.jdbc.Driver"
}
For CSV Files
The following are the source and destination attributes for CSV files:
| Attribute | Default Value | Mandatory/Optional | Description |
|---|---|---|---|
| @type | No default | Mandatory | CSV_FILE. |
| filepath | No default | Mandatory | The filepath relative to the inputDir or outputDir path in the config file. |
| delimiter | No default | Mandatory | The character that is used to divide one column from the next in the input file. Any prinatble ASCII character (except for backslash ) can be used as a delimiter, in addition to space ( ), tab, and Ctrl-A. Note: The delimiter and qualifier characters cannot be the same. |
| qualifier | No default | Optional | Character to enclose fields that contain a delimiter character. Any printable ASCII character (except for backslash ) can be used as a qualifier. This is used when source is a file and the delimiter character is itself a part of the data to be transformed. So to tell BDT that this is not a delimiter, user can enclose input data by a qualifier and mention qualifier here in this field. Note: The delimiter and qualifier characters cannot be the same. |
| columnCount | No default | Mandatory | Number of columns in the input file. This does not necessarily equal the number of columns that have rules defined for them in this bdt.policy file. |
| hasHeaderRow | false | Optional | True or false. Default false. If true, the utility will treat the first row as header row. Values from first row will be treated as column names. The same column names should be mentioned in the transformation action. If false, the utility will treat all the rows as data to be transformed. |
| unescapeInput | false | Optional | True or false. Default: false. If true, delimiter and qualifier characters in input preceded by a backslash (\) are considered as normal characters and part of the input data. If you are using the BDT output file as input to another process, set unescapeInput to true. |
| encoding | UTF8 | Optional | Used to set character encoding, for example, to enable IBM’s EBCDIC. Input file must use same character encoding. Output and failed record files will be created with the same character encoding. Sample: For EBCDIC encoding, set encoding:"ibm500" in the policy file.To encrypt ASCII files, you must set encoding: ASCII. Refer to Supported Encodings for details. |
| allowSingleCharInputs | false | Optional | True or false. Default is false. Applicable to FPE and FF1 only. If true, single character inputs are passed through untransformed, but otherwise transformation continues. If false, row transformation fails for single character inputs. |
Note
For file-to-file (CSV) transformation, if the delimiter or qualifier is not configured for destination, the delimiter or qualifier configured in the source will be considered.
For Fixed-Length Files
The following are the source and destination attributes for fixed-length files:
| Attribute | Default Value | Mandatory/Optional | Description |
|---|---|---|---|
| @type | No default | Mandatory | FIXED_LENGTH_FILES. |
| filepath | No default | Mandatory | The filepath relative to inputDir or outputDir path in config file. |
| columnCount | No default | Mandatory | The number of columns a row has. |
| encoding | No default | Optional | Used to set character encoding, for example, to enable IBM’s EBCDIC. Input file must use same character encoding. Output and failed record files will be created with the same character encoding. Sample: For EBCDIC encoding, set encoding:" ibm500" in the policy file.Refer to Supported Encodings for details. |
| columnPosition Info | No default | Mandatory | Column positioning. |
| recordLength | No default | Mandatory | Record length. |
| lineSeparator | false | Optional | True or false. Is record separated with line separator? |
| hasHeaderRow | false | Optional | True or false. Default false. If true, the utility will treat the first row as header row. Values from this row will be treated as column names. The same column names should be mentioned in the transformation action. |
Tables Attributes
| Attribute | Default Value | Mandatory/Optional | Description |
|---|---|---|---|
| sourceTable | No default | Mandatory | Source table name. For database only. Refer to Limitations for details. |
| destinationTable | No default | Mandatory (if inPlaceUpdate attribute is set to false) | Destination table name. For database only. If the policy file defines an in-place transformation of data (with inPlaceUpdate attribute set to true), the destination value is ignored if defined. By definition, source and destination tables are the same. |
| columns | No default | Mandatory | List of columns to be transformed and transformation operation details. Policy rules for transforming one or more columns of data in the input file. For each column where transformation is desired, the bdt.policy file specifies the data transformation action to perform and a set of parameters related to that action. |
| name | No default | Mandatory | Column name Note: Applicable for File to DB Transformation only. If hasHeaderRow is set to false, then configure this parameter as "columnN" where N is the position of column.Note: The position of column is starting from 0. For example: If the user wants to transform first column, then the value will be "column0", for second column; the value will be "column1", and so on. |
| action | No default | Mandatory | Action to be taken. Supported options: ENCRYPT, DECRYPT, TOKENIZE, DETOKENIZE, REKEY. |
| Config@type | No default | Mandatory | Options: AES_CBC_PAD, AES_CTR, FPE, FF1, TOKENIZE, DESede (signifying the triple operations of DES of encrypt-decrypt-encrypt).To encrypt/decrypt data using either of the NIST-defined FPE formats — FF3 or FF1 — the user must specify the ENCRYPT or DECRYPT policy actions with a type value. The value for FF1 is FF1. The value for FF3 is FPE. Keep in mind that the BDT syntax uses FPE as a synonym for FF3. |
| key | No default | Mandatory | Name of the encryption key. |
| header | No default | Optional | Cipher header version for determining key version. Supported header versions: V1_5, V1_5_Base64, V2_1, and V2_7. |
| tweak | No default | Mandatory (if Config@type is FPE or FF1) | There are two ways of defining tweak:"tweak":<16 characters long HEX encoded string>"tweak":"auto"Either of these ways can be used. Refer to the tweakSource attribute for details. |
| tweakSource | No default | Optional | Specifes a column in the source for the tweak, to allow different tweaks to be used for each row.For example: "tweakSource":"columnName".tweakSource is not supported when inPlaceUpdate is set to true. |
| mode | No default | Mandatory (if Config@type is DESede) | Specify the supported algorithm mode, either CBC or ECB. The mode must be specified when using the DESede (that is, the 3DES) algorithm (as specified by the @type attribute).DESede means DES encrypt-decrypt-encrypt, a common synonym for 3DES that summarizes the basic three-step process underlying it. Note: ECB mode is not recommended. |
| padding | No default | Mandatory (if Config@type is DESede) | Valid options are PKCS5Padding or NoPadding. |
| iv | No default | Optional | Initialization Vector. Refer to supported encryption algorithms for details. Note: It is recommended to specify random generated IV, and do not reuse any IV. |
| ivSource | No default | Optional | Specifes a column in the source for the IV, to allow different IVs to be used for each row.For example: "ivSource":"columnName".Value to be specified in hexadecimal format. For example: 01A40933B401A409. If value is not valid hexadecimal, then it is treated as an ASCII string. ivSource is not supported when inPlaceUpdate is set to true.Note: It is recommended to specify random generated IV, and do not reuse any IV. |
| characterSet | No default | Mandatory (if Config@type is FF1 or FPE) | Name of the characterSet that was defined under characterSets in the bdt.config file. Defaults include "latin", "digits", "alphabets", and "alphanumeric".Users can also define their own characterSet in the config file. |
| subset | No default | Optional | Table-specific data filter criteria. Filter input data based on these criteria. |
| offset | 0 | Optional | Start record index. |
| limit | No default | Mandatory (if filtering data for transformation) | Maximum number of records/rows to transform, starting from offset. |
| recurrency | 1 | Optional | Select every Nth record. Default is 1 (process every record). |
| filters | No default | Mandatory (if filtering data for transformation) | Subset-specific filters to be used. |
| selector | No default | Mandatory (if filtering data for transformation) | Column name. |
| operator | No default | Mandatory (if filtering data for transformation) | Options: "=","<",">","<=",">=","!=","like". |
| expression | No default | Mandatory (if filtering data for transformation) | Expression to evaluate filter on selector using specified operator. |
| orderBy | First column | Optional | Order by column, for example: column1. Not supported for file transformation. |
| order | ASC | Optional | ASC (ascending) or DESC (descending) record order. Not supported for file transformation. |
| createDestinationTable | false | Optional | True or false. Create destination table if none exists. If in-place transformation is defined (when inPlaceUpdate attribute is set to true), createDestinationTable is ignored, because no new destination table is created. For File to DB, datatype of columns in the destination table will be varchar(10000).Note: The createDestinationTable attribute is not supported for HANA databases.Caution: Use only for testing. Do not use in a production environment. |
| inputEncoding | No default | Optional | Specifies how to decode input and create byte array from it. Valid options are BASE2, BASE16, BASE64, UTF8, UTF16LE, UTF16BE, UTF32LE, or UTF32BE.Note: BASE2 is supported only with database Blob data type columns. |
| outputEncoding | No default | Optional | Specifies how to encode output byte array to a string. Valid options are BASE16, BASE64, UTF8, UTF16LE, UTF16BE, UTF32LE, or UTF32BE. |
| inPlaceOnError | No default | Optional | Specifies what to do when some records are failed. Only valid when in-place transformation is defined (inPlaceUpdate attribute is true). Values are rollback, commit, or exit.If rollback, roll back all changes. Source table returns unchanged to its initial state.If commit, the source column is updated with transformed values, and for any failed records, commit with null or plain text value as specified by the errorPlaceHolder setting.If exit, do not commit or rollback, but keep source table as it is, so that you can resolve the failure cause and restart the transformation (with -r option). |
| errorPlaceHolder | No default | Mandatory (if inPlaceOnError is set to commit) | NULL or PLAIN_TEXT.If Null, set column value to null if transformation fails.If plain_text, the column retains its original plain text if transformation fails.Caution: If using Null value, it could cause loss of plaintext data. |
The following is a sample policy file excerpt for table attributes (Subset, Offset, Limit, Filters, Selector, Operator, Expression) meant for filtering input data:
"subset":{
"offset":"10",
"limit":1000,
"filters":[
{
"selector":"salary",
"operator":">",
"expression":"10000"
},
{
"selector":"name",
"operator":"like",
"expression":"'p%'"
}
]
}
Operations Parameters
For each column of input data, the bdt.policy file can specify one of the following operations:
Note
If you do not specify any of these actions for a column, the contents of the column are copied to the output file without any transformation.
