Your suggested change has been received. Thank you.

close

Suggest A Change

https://thales.na.market.dpondemand.io/docs/dpod/services/kmo….

Thales Logo

All

  • All
back

Tasks

How to protect data

search

Please Note:

How to protect data

This section outlines the complete File-to-File transformation process for CSV files. The example provided here demonstrates how to protect data of a CSV file. The protect operation protects the plaintext data and returns the ciphertext based on the protection policy selected in the job configuration. The protection policy provides different functionalities as described here; the output will change according to these functionalities.

Important Points

  • We have provided the following sample files with this example.

    • Input.csv: Contains the data to be processed. Download and save this file on the host machine specified during volume mapping when deploying BDT.

    • Output.csv: Contains the processed data. This file will get created on the host machine specified during volume mapping when deploying BDT.

  • Column of the input.csv file that to be protected: latitude.

  • Type of protection policy used: Internal version.

  • Algorithm: FPE/FF1v2.

  • Character set: Alphanumeric.

Prerequisites

  • BDT application must be registered on the CipherTrust Manager. Refer to Quick Start for details.

  • BDT must be deployed. Refer to Quick Start for details.

  • A compatible CipherTrust Manager must be up and running. Refer to CipherTrust Manager Deployment for details.

Steps

The transformation steps provided in this example can be performed using the CipherTrust Manager UI or API playground.

  1. Create source and destination data-sources. The source data-source contains the data to be processed while the destination data-source will store the processed data.

    API: post /v1/data-protection/data-sources

    Create source data-source

    Request
        {
    "name": "sourcefile",
    "description": "source details",
    "type": "CSV File",                    
    "file_properties":

    {"file_path": "/opt/app/tmp/input/inputProtect.csv", "delimiter": ",",   "column_count": 12, "has_header_row": true}
    }
    Response
        {
        "id": "5c22a447-f8dd-4c4c-af7d-149123ecf974",
        "uri": "kylo:kylo:adp-management:data-sources:sourcefile-5c22a447-f8dd-4c4c-af7d-149123ecf974",
        "account": "kylo:kylo:admin:accounts:kylo",
        "createdAt": "2025-03-20T11:08:05.940999Z",
        "updatedAt": "0001-01-01T00:00:00Z",
        "name": "sourcefile",
        "description": "source details",
        "type": "CSV File",                        
        "file_properties": {
            "file_path": "/opt/app/tmp/input/input.csv",
            "delimiter": ",",
            "qualifier": "\"",
            "column_count": 12,
            "has_header_row": true,                           
            "encoding": null,
            "record_length": null,
            "line_separator": null
        }
    }

    In response, ID of sourcefile (5c22a447-f8dd-4c4c-af7d-149123ecf974) is returned.

    Create destination data-source

    Request
        {
    "name": "destinationfile",
    "description": "destination details",
    "type": "CSV File",                    
    "file_properties":

    {"file_path": "/opt/app/tmp/output/outputProtect-2025-03-26.10.35.15.348.csv", "delimiter": ",", "column_count": 12,"has_header_row": true}
    }
    Response
        {
        "id": "9b9520e2-2d86-46bd-a615-375b689ee03b",
        "uri": "kylo:kylo:adp-management:data-sources:destinationfile-9b9520e2-2d86-46bd-a615-375b689ee03b",
        "account": "kylo:kylo:admin:accounts:kylo",
        "createdAt": "2025-03-20T10:56:19.706092Z",
        "updatedAt": "0001-01-01T00:00:00Z",
        "name": "destinationfile",
        "description": "destination details",
        "type": "CSV File",                        
        "file_properties": {
            "file_path": "/opt/app/tmp/output/output.csv",
            "delimiter": ",",
            "qualifier": "\"",
            "column_count": 12,
            "has_header_row": true,                            
            "encoding": null,
            "record_length": null,
            "line_separator": null
        }
    }

    In response, ID of destinationfile (9b9520e2-2d86-46bd-a615-375b689ee03b) is returned.

    To create a data source from CipherTrust Manager's UI, follow this link.

    Similarly, you can also create data sources for fixed length file and databases. For fixed length file follow the link. For databases, refer to Managing Data Sources.

  2. Create access policy

    API: /v1/data-protection/access-policies

    Request
        {
    "name": "access_policy",
    "description": "access policy ",
    "default_reveal_type": "Plaintext"
    }
    Response
        {
        "id": "24e1f7eb-63a8-4b9f-abc3-0295098ad98f",
        "uri": "kylo:kylo:adp-management:access-policies:access-policy-24e1f7eb-63a8-4b9f-abc3-0295098ad98f",
        "account": "kylo:kylo:admin:accounts:kylo",
        "createdAt": "2025-03-18T11:30:54.511291Z",
        "updatedAt": "2025-03-18T11:30:54.513016Z",
        "created_by": "local|238dfbe5-dff4-4c56-8474-480fb3c51701",
        "name": "access_policy",
        "description": "",
        "default_reveal_type": "Plaintext",
        "default_error_replacement_value": "",
        "default_masking_format_id": null,
        "user_set_policy": [],
        "version": 1
    }

    In response, ID of the access policy (24e1f7eb-63a8-4b9f-abc3-0295098ad98f) is returned.

    To create access policy from CipherTrust Manager's UI, follow this link.

  3. Create protection policy.

    API: post /v1/data-protection/protection-policies

    API Request
        {
    "name": "fpeProtectionPolicy",
    "description": "FPE protection policy",
    "key": "aakey",
    "algorithm": "FPE/FF1v2/UNICODE",
    "tweak": "121212121212",
    "tweak_algorithm": "SHA1",
    "character_set_id": "2d00cfdb-c422-46ac-8a05-6152806a5715",
    "access_policy_name": "access_policy"
    }
    API Response
        {
        "id": "ed2efc0b-bf78-4b0f-bb0a-2077475fe5ba",
        "uri": "kylo:kylo:adp-management:protection-policies:fpeprotectionpolicy-v10-ed2efc0b-bf78-4b0f-bb0a-2077475fe5ba",
        "account": "kylo:kylo:admin:accounts:kylo",
        "createdAt": "2025-04-02T06:29:16.772448Z",
        "updatedAt": "2025-04-02T06:29:16.771811Z",
        "name": "fpeProtectionPolicy",
        "version": 10,
        "latest_version": true,
        "description": "",
        "key": "aakey",
        "iv": "",
        "tweak": "121212121212",
        "tweak_algorithm": "SHA1",
        "character_set_id": "2d00cfdb-c422-46ac-8a05-6152806a5715",
        "character_set": {
            "id": "2d00cfdb-c422-46ac-8a05-6152806a5715",
            "uri": "kylo:kylo:adp-management:character-sets:alphanumeric-2d00cfdb-c422-46ac-8a05-6152806a5715",
            "account": "kylo:kylo:admin:accounts:kylo",
            "createdAt": "2025-01-31T07:15:53.167752Z",
            "updatedAt": "2025-01-31T07:15:53.167752Z",
            "name": "Alphanumeric",
            "description": "",
            "range": "0030-0039,0041-005A,0061-007A",
            "encoding": "UTF-8",
            "predefined": true
        },
        "masking_format_id": "",
        "algorithm": "FPE/FF1v2/UNICODE",
        "use_external_versioning": false,
        "disable_versioning": false,
        "access_policy_name": "access_policy",
        "prefix": "",
        "data_format": "",
        "nonce": "",
        "tag_length": 0,
        "allow_small_input": true
    }

    To create protection policy from CipherTrust Manager's UI, follow this link.

    A protection policy, named fpeProtectionPolicy is created and this policy will be used to protect data as shown the in next step.

    Note

    Ensure the key used in protection policy has correct permissions. For more details on key permissions, refer to Supported key types.

  4. Create job configuration.

    API: post /v1/data-protection/bdt/job-configurations

    API Request
        {
    "name": "job configuration for file",
    "in_place_update": false,
    "source_id": "5c22a447-f8dd-4c4c-af7d-149123ecf974",
    "destination_id": "9b9520e2-2d86-46bd-a615-375b689ee03b",
    "tables": [
        {
        "columns": [

    { "source_column_name": "latitude", "operation": "protect",     "protection_policy": "fpeProtectionPolicy"}
        ]
        }
    ]
    }
    API Response
        {
        "id": "142b84d8-22ba-4b66-8fec-16076911ec7a",
        "uri": "kylo:kylo:adp-management:bdt-job-configurations:job-configuration-for-file-142b84d8-22ba-4b66-8fec-16076911ec7a",
        "account": "kylo:kylo:admin:accounts:kylo",
        "createdAt": "2025-03-18T11:43:40.577668Z",
        "updatedAt": "2025-04-04T09:14:59.905281Z",
        "version": 26,
        "name": "job configuration for file",
        "description": "",
        "case_sensitive": false,
        "source_id": "5c22a447-f8dd-4c4c-af7d-149123ecf974",
        "destination_id": "9b9520e2-2d86-46bd-a615-375b689ee03b",
        "in_place_update": false,
        "unprocessed_record_file": "/opt/app/tmp/unProcessedRecordFiles/",
        "tables": [
            {
                "id": "2eefe155-7bee-41a9-aacf-546c52f75880",
                "uri": "kylo:kylo:adp-management:bdt-job-configurations-tables:2eefe155-7bee-41a9-aacf-546c52f75880",
                "account": "kylo:kylo:admin:accounts:kylo",
                "createdAt": "2025-04-04T08:11:39.328142Z",
                "updatedAt": "2025-04-04T08:11:39.327642Z",
                "source_table": "",
                "source_schema": "",
                "destination_table": "",
                "destination_schema": "",
                "create_destination_table": false,
                "subset": {
                    "limit": 0,
                    "offset": 0,
                    "recurrency": 1,
                    "order_by": "",
                    "order": "ASC",
                    "filters": []
                },
                "columns": [
                    {
                        "id": "dfd71200-1c83-40c5-921d-e941496a72b2",
                        "uri": "kylo:kylo:adp-management:bdt-job-configurations-tables-columns:dfd71200-1c83-40c5-921d-e941496a72b2",
                        "account": "kylo:kylo:admin:accounts:kylo",
                        "createdAt": "2025-04-04T08:11:39.332823Z",
                        "updatedAt": "2025-04-04T08:11:39.332262Z",
                        "source_column_name": "latitude",
                        "operation": "protect",
                        "protection_policy": "fpeProtectionPolicy"
                    }
                ]
            }
        ],
        "meta": {
            "job_running": false
        }
    }

    In response, ID of the job configuration (142b84d8-22ba-4b66-8fec-16076911ec7a) is returned.

    To create job configuration from CipherTrust Manager's UI, follow this link.

    To protect, select operation type as protect.

    Tip

    Multiple columns can be configured in a single job configuration, and each column can be associated with different protection policies.

  5. Run job.

    To run a job, at least one active client must be registered with the application.

    There is slight difference in the Run Job API call depending on the CipherTrust Manager version you are using. Choose the preferred tab to view the details.

    API: post /v1/data-protection/run-job

    API Request
        {
    "job_configuration_id": "142b84d8-22ba-4b66-8fec-16076911ec7a",
    "client_profile_id": "9f3edcb1-38e8-4bc7-8c41-99e897560328"
    }
    API Response
        { "job_status_id": "a960a20c-4852-4fee-a18b-302710f67170"}

    In response, a job status ID (a960a20c-4852-4fee-a18b-302710f67170) is returned, use this ID to check the intermediate status of the job.

    API: post /v1/data-protection/bdt/job-configurations/{id}/run

    API Request
    post /v1/data-protection/bdt/job-configurations/{142b84d8-22ba-4b66-8fec-16076911ec7a}/run            
{ "client_profile_id": "9f3edcb1-38e8-4bc7-8c41-99e897560328"}
    API Response
        { "job_status_id": "a960a20c-4852-4fee-a18b-302710f67170"}

    In response, a job_status_id (a960a20c-4852-4fee-a18b-302710f67170) is returned. Use this ID to check the status of the job.

    To run a job from CipherTrust Manager's UI, follow this link.

  6. Check status of the job.

    There is slight difference in the Job Status API call depending on the CipherTrust Manager version you are using. Choose the preferred tab to view the details.

    API: get /v1/data-protection/job-status/{id}

    Pass the job_status_id (generated in step 5) in the API request.

    API Request
        {
    /v1/data-protection/job-status/{a960a20c-4852-4fee-a18b-302710f67170} 
    }

    API: get /v1/data-protection/jobs/{id}

    Pass the job_status_id (generated in step 5) in the API request.

    API Request
    /v1/data-protection/jobs/a960a20c-4852-4fee-a18b-302710f67170

    To run a job from CipherTrust Manager's UI, follow this link.

    In response, information about the jobs in various stages, total number of records processed, error messages are returned.

Verify Results

After the job status gets completed, check the Output.csv for transformed data.

Note

For transformations, where destination is file, the output file name (provided on UI) will also contain a timestamp appended to it. For example, <output-name-as-per-ui>-2025-04-01.17.57.01.521.csv

  • For CSV file, the output file will be formatted as: <file-name>-<timestamp>.csv

  • For fixed length file, the output file will be formatted as: <file-name>-<timestamp>.txt

In the Input.csv file, the latitude column was in plaintext, after transformation, the data is in ciphertext.

Before TransformationAfter Transformation
Inputoutput

In the output, you can see that the version header is prepended to the ciphertext (1002008eb.QbvK1B).

Protect Use cases

Protect with Prefix

To configure prefix, enter a user friendly name in the Prefix text-box when configuring protection policy.

Example

Protection Policy Type: Internal version

Prefix: geo-

Character Set: Alphanumeric

Input data: 38.689999

Ciphertext: 1002008geo-eb.QbvK1B

Protect Luhn comaplaint data

To protect luhn compliant data, the Luhn toggle must be turned on when configuring protection policy. To use, this functionality, following points must be considered:

  • It is only compatible with All digits character set (0-9) and FPE algorithms.

  • The luhn check requires minimum3 characters to perform crypto operations.

Example

Protection Policy Type: Internal version

Character Set: All digits (0030-0039)

Input data: 1234567812345670

Ciphertext: 9262746894652355

Note

The generated ciphertext is also luhn complaint. To check if a number is valid luhn, starting from the rightmost digit, double every other digit, and if the doubled value is greater than 9, add its digits. Then, sum all the digits (including the original and doubled ones) and check if the sum is divisible by 10.

Protect with static masking format

The static masking format allows you to preserve starting and ending characters of the input data. It is only applicable with FPE algorithms.

To configure static masking format, select the desired format when configuring protection policy.

Example

Protection Policy Type: Internal version

Static masking format: Preserve LAST_FOUR characters

Character Set: Alphanumeric (0030-0039, 0041-005A, 0061-007A)

Note

Any character outside the given character set range will be preserved in the output.

Input Data: 3526 HIGH ST

Ciphertext:10020080j37 PaGH ST (last 4 characters are preserved.)

On this page