Recovering a Failed or Incomplete dataxform Session

A dataxform session can fail for different reasons: dataxform can be canceled, the process is killed, the system crashes, and so on. The recovery process is similar for manual and automatic dataxform.

For more information, see:

• Restarting an Incomplete Automatic dataxform Session

• Recovering from a Failed dataxform Session

• Avoiding Double Encryption

Restarting an Incomplete Automatic dataxform Session

Although unlikely, if an automatic dataxform session fails to complete, your intervention may be required to allow it to resume.

In the case of a system restart--for example after a power failure--the automatic session will be resumed when the GuardPoint is mounted during the boot sequence. However, if the dataxform session terminated unexpectedly (for example, if the session had been killed), then the system will not restart it automatically, and the GuardPoint status needs to be reset so that another automatic session will be initiated.

The following steps reset the GuardPoint status:

1. Verify that the session really has terminated unexpectedly by verifying that no existing dataxform process is running. Use the standard system tools (ps or task manager).

2. From your key manager, disable the GuardPoint, then wait until the GuardPoint status on the CTE Agent no longer lists the GuardPoint.

3. From your key manager enable the GuardPoint.

An automatic dataxform session will start soon after this, and the session will continue from where it previously stopped.

Recovering from a Failed dataxform Session

The following is an example of how to recover from a failed dataxform session. The GuardPoint in this example is /opt/apps/dx9.

1. A manual dataxform session is run, then canceled using Ctrl-c:

   # dataxform --rekey --nq --print_stat --preserve_modified_time --gp /opt/apps/dx9
   Checking if data transform is supported for guard point /opt/apps/dx9
   Data transformation is supported on /opt/apps/dx9
   About to perform the requested data transform operation
   -- Be sure to back up your data
   -- Do not access files in the guard point during the transform process
   -- Please do not attempt to terminate the application
   Scan found 10003 files (273 KB) in 5 directories for guard point /opt/apps/dx9
   The current operation took 0 hours, 0 minutes and 2 seconds
   Shutting down data transform: received fatal signal 2
   Transformed 1126 files (24 KB) of 10003 files (273 KB) for guard point /opt/apps/dx9
   The current operation took 0 hours, 0 minutes and 7 seconds
   Data transform got errors on some files
   The file /opt/apps/dx9/datafile931 could not be transformed, a signal stopped the data transform
   The file /opt/apps/dx9/datafile1102 could not be transformed, a signal stopped the data transform
   The file /opt/apps/dx9/datafile1121 could not be transformed, a signal stopped the data transform
   The file /opt/apps/dx9/datafile1100 could not be transformed, a signal stopped the data transform
   The file /opt/apps/dx9/datafile1120 could not be transformed, a signal stopped the data transform
   The file /opt/apps/dx9/datafile1132 could not be transformed, a signal stopped the data transform
   Number of files in error due to a signal stopping the dataxform: 6
   The data transform operation took 0 hours, 0 minutes and 7 seconds
   Could not complete data transform for guard point /opt/apps/dx9, data transform was interrupted by a signal
   Data transform for guard point /opt/apps/dx9 finished but 6 files were not processed due to errors
   

   The dataxform_status-_opt_apps_dx9, dataxform_status_error-_opt_apps_dx9, and dataxform_dir_list-_opt_apps_dx9 files are created in the log directory, or the GuardPoint, depending on the command line options. By default, all status and log files go to /var/log/vormetric. If --status_gp was included on the command line, the status files go in the GuardPoint.

   The dataxform session log file, /var/log/vormetric/vordxf-_opt_apps_dx9_root.log, is updated. It displays the same basic information as displayed on the terminal screen.

2. The dataxform messages indicate that the session had been interrupted and that a number of files (6) are in an unknown state due to the interruption.

3. You may optionally use the --recovery option to generate a set of files that track the progress dataxform made in the GuardPoint (see later). However this step is not required and can be deferred until after a new dataxform session has been run to transform the remaining files.

4. Resume the data transformation run, using the same command as you used to start it before.

   # dataxform --rekey --nq --print_stat --preserve_modified_time --gp /opt/apps/dx9
   Checking if /opt/apps/dx9 is a guard point with a rekey policy applied /opt/apps/dx9 is a guard point with a rekey policy applied
   Automatic data transform status for /opt/apps/dx9: previous attempt did not complete
   Note: data from a previous dataxform run is being used
   About to perform the requested data transform operation
   -- Be sure to back up your data
   -- Please do not attempt to terminate the application
   Scan found 10003 files (273 KB) in 5 directories for guard point /opt/apps/dx9
   Transformed 10001 files (273 KB) of 10003 files (273 KB) for guard point /opt/apps/dx9
   Data transform got errors on some files
   The file /opt/apps/dx9/datafile931 could not be transformed, it was in progress in the previous data transform
   The file /opt/apps/dx9/datafile1102 could not be transformed, it was in progress in the previous data transform
   The file /opt/apps/dx9/datafile1121 could not be transformed, it was in progress in the previous data transform
   The file /opt/apps/dx9/datafile1100 could not be transformed, it was in progress in the previous data transform
   The file /opt/apps/dx9/datafile1120 could not be transformed, it was in progress in the previous data transform
   The file /opt/apps/dx9/datafile1132 could not be transformed, it was in progress in the previous data transform
   The data transform operation took 0 hours, 0 minutes and 7 seconds
   Could not complete data transform for guard point /opt/apps/dx9, data transform was interrupted by a signal
   Data transform for guard point /opt/apps/dx9 finished but 6 files were not processed due to errors
   

5. If for some reason this session also did not complete, performing the same operation again should continue from where the previous session ended. The status records are accumulated across each session.

6. Use the --recovery option to generate a set of files that track the progress dataxform made in the GuardPoint. Use these files later to determine which files in the GuardPoint have not been transformed.

   Do not run the --recovery option more than once, as a second run may overwrite vital information required to recover any files that did not transform correctly.

   # dataxform --recovery --gp /opt/apps/dx9
   Note: data from a previous dataxform run is being used
   Number of files previously in error due to a signal stopping the dataxform: 6
   Scan found 10010 files (273 KB) in 6 directories for guard point /opt/apps/dx9
   Generating list of files previously transformed on /opt/apps/dx9
   Data transform got errors on some files
   The file /opt/apps/dx9/datafile931 was previously in error. A signal stopped the dataxform
   The file /opt/apps/dx9/datafile1102 was previously in error. A signal stopped the dataxform
   The file /opt/apps/dx9/datafile1121 was previously in error. A signal stopped the dataxform
   The file /opt/apps/dx9/datafile1100 was previously in error. A signal stopped the dataxform
   The file /opt/apps/dx9/datafile1120 was previously in error. A signal stopped the dataxform
   The file /opt/apps/dx9/datafile1132 was previously in error. A signal stopped the dataxform
   Number of files in error due to a signal stopping the dataxform: 6
   The dataxform_files_todo-_opt_apps_dx9 and dataxform_files_done-_opt_apps_dx9 are created, and the /var/log/vormetric/dataxform_status-_opt_apps_dx9 file may be updated.
   
   # cat dataxform_status-_opt_apps_dx9
   version=5
   status=done
   operation=rekey
   current=
   0 in-progress files
   seqno=2
   hmac=5449453CBAEFCC01EC02542650D8C1040D762D213829F8B3CF967DC578320A475F705C11D3BAF74D588630CE8078AF46
   
   This file indicates that the dataxform session had completed.
   

7. The /var/log/vormetric/vordxf-_opt_apps_dx9_root.log file is updated. It displays the same basic information displayed on the terminal screen.

8. The primary files of interest are dataxform_files_todo-_opt_apps_dx9, dataxform_files_done-_opt_apps_dx9, and dataxform_status_error-_opt_apps_dx9.

   dataxform_files_todo-_opt_apps_dx9 lists the files that dataxform had not touched and are yet to be transformed, and any files for which a transform was attempted but for some reason failed.

   dataxform_files_done-_opt_apps_dx9 lists the files that dataxform rekeyed successfully. Leave these files alone.

   dataxform_status_error-_opt_apps_dx9 lists the files that were being processed at the time an error occurred—for example, when dataxform was interrupted in the above example. These are the files that have to be checked individually to determine if they had been processed. They may or may not have been completely processed.

   # cat dataxform_status_error-_opt_apps_dx9
   Error, was in progress during a previous session: /opt/apps/dx9/datafile931
   Error, was in progress during a previous session: /opt/apps/dx9/datafile1102
   Error, was in progress during a previous session /opt/apps/dx9/datafile1121
   Error, was in progress during a previous session: /opt/apps/dx9/datafile1100
   Error, was in progress during a previous session: /opt/apps/dx9/datafile1120
   Error, was in progress during a previous session: /opt/apps/dx9/datafile1132
   
   **Note**: If more than one session restart was required to complete the dataxform run, there may be repeated entries in the above list.
   {.note}
   
   # cat dataxform_files_todo-_opt_apps_dx9
   /opt/apps/dx9/datafile931
   /opt/apps/dx9/datafile1100
   /opt/apps/dx9/datafile1102
   /opt/apps/dx9/datafile1120
   /opt/apps/dx9/datafile1121
   /opt/apps/dx9/datafile1132
   
   # head -12 dataxform_files_done-_opt_apps_dx9
   /opt/apps/dx9/datafile1
   /opt/apps/dx9/datafile2
   /opt/apps/dx9/datafile3
   /opt/apps/dx9/datafile4
   /opt/apps/dx9/datafile5
   /opt/apps/dx9/datafile6
   /opt/apps/dx9/datafile7
   /opt/apps/dx9/datafile8
   /opt/apps/dx9/datafile9
   /opt/apps/dx9/datafile10
   /opt/apps/dx9/datafile11
   /opt/apps/dx9/datafile12
   

9. Disable the GuardPoint with the rekey policy through your key manager. Do not re-apply the regular policy because you are not able to use the in the GuardPoint. A proper rekey policy restricts access to all the files in the GuardPoint. You have to disable the rekey policy so you can access all the files in the GuardPoint and determine their transformation status.

10. At this point, you should restore the files that were named in the error listing above from a backup, and run a transformation session for just these files, using the "todo" list (dataxform_files_todo-_opt_apps_dx9) to select which files are to be transformed. However, depending on the exact nature of the error reported, some entries may need to be removed from the "todo" list, for example, if somehow a file no longer exists, then attempting to transform it again will obviously not work.

11. If you are running automatic dataxform, remove the dataxform_auto_conf file from the GuardPoint.

12. Re-apply the rekey policy to the GuardPoint, but first be sure that your current directory is not the GuardPoint.

13. Verify that the policy has been successfully re-applied.

    # secfsd -status guard
    GuardPoint Policy Type ConfigState Status Reason
    ---------- ------ ---- ----------- ------ ------
    /opt/apps/dx1 allowAllOps_fs local guarded guarded N/A
    /opt/apps/dx3 denyAllOps_fs local guarded guarded N/A
    /opt/apps/dx4 allowAllOps_fs local guarded guarded N/A
    /dev/dsk/c0t0d0s7 allowAllOps_rd rawdevice guarded guarded N/A
    /opt/apps/dx9 clear_to_aes128_dx local guarded guarded N/A
    

14. Run dataxform on just the files in the todo list. For example:

    # dataxform --rekey_list --file_list var/log/vormetric/dataxform_files_todo-_opt_apps_dx9 --nq --print_stat --preserve_modified_time --gp /opt/apps/dx9
    Checking if data transform is supported for guard point /opt/apps/dx9
    Data transformation is supported on /opt/apps/dx9
    Previous status information does not relate to a --rekey_file operation.
    About to perform the requested data transform operation
    -- Be sure to back up your data
    -- Please do not access files in the guard point during the
    transform process
    -- Please do not attempt to terminate the application
    Starting data transform of /opt/apps/dx9 for files listed in
    /var/log/vormetric/dataxform_files_todo-_opt_apps_dx9
    The data transform operation took 0 hours, 1 minutes and 20 seconds
    bash-3.00#
    

    In this case the dataxform completes successfully. The error file was removed because no errors were encountered in the rekey process.

15. Disable the rekey policy.

16. If you are satisfied with the successful completion, clean up the files left by the rekey process.

    # dataxform --cleanup --gp /opt/apps/dx9
    About to remove the data transformation status files
    Do you wish to continue (y/n)? **y**
    

17. Apply a regular policy that uses the applied key.

18. Check that the access controls and encryption keys configured in the policy are working as expected.

Avoiding Double Encryption

Scenarios can occasionally occur that can lead to double-encryption. This can be caused by various different issues such as:

• Attempts to run Data Transformation twice on the same content, using the same policy, so on the second transformation it attempted to encrypt the file with the target key, not the source key

• Bad restore attempt

• Files incorrectly copied into a GuardPoint

• Procedural problem with encrypting the GuardPoint

• The file is unencrypted, or encrypted without a header, and the policy assumes that the file must be encrypted with a header

• The file is corrupted, in which case you need to recover the file from a backup method

• The wrong transform policy was used, so you need to change the policy

CipherTrust Transparent Encryption Userspace has added two error messages to help avoid the double-encryption scenario. They are:

• Invalid header

• Wrong key

Invalid Header

If Data Transformation is converting from an encrypted state and its header has been corrupted, the following message displays as the file is being processed:

    The file (filename) has an invalid header; skipping

At the end of the dataxform run, it will display another message for each file skipped, and a count of the number of files skipped:

    The file (filename) was skipped. It has an invalid header
    Number of files skipped due to an invalid header: (count)

Encrypted with Incorrect Key

If converting from an encrypted state and the encryption key of a file does not match the initial key in the transform rule, the following message displays as the file is being processed:

    The file (filename) was skipped. It is encrypted with a wrong key

At the end of the dataxform run, it displays another message for each file skipped, and a count of the number of files skipped:

    The file (filename) was skipped. It is encrypted with a wrong key
    Number of files skipped due to a wrong key: (count)