Troubleshooting
Error Messages
This section lists the various error messages that the system can display, with explanations and solutions (if available).
Locations
| Error Message | Explanation |
|---|---|
| "Branch Location name already exists" (message on toast) | You tried to create a location with a name that is already taken by another location. SOLUTION: Choose another name. |
Data Stores
| Error Message | Explanation |
|---|---|
| "Data Store name already exists" (message on toast) | You tried to create a data store with a name that is already taken by another data store. SOLUTION: Choose another name. |
| "A valid agent could not be found" (Agent selection - on mouse-over on the data store) | There is no active agent for this data store that the automatic agent selection process has been able to detect. SOLUTION: This requires additional research, such as checking if the agent is installed on the data store, if it has the right type (local/proxy), if it is of the right OS “flavor” (Linux, Windows), or of the right type (e.g. database). Refer to the Data Discovery and Classification Deployment Guide for more information on troubleshooting this issue. |
| "One of the Oracle Data Stores has been configured with a wrong schema." | SOLUTION: Check the Oracle schema that you used and change it if necessary. For more information, see Add Oracle Data Store. |
| "The target for Data Store SharePoint scan for sites does not have access permissions" | If DDC returns this error while adding a SharePoint Online data store, it is because “grant app permission” is disabled by default on SharePoint Online. SOLUTION: For the SharePoint Add-In to work, the DisableCustomAuthenticationApp setting for the tenants needs to be set to "false". Follow the steps in Configure SharePoint Online. |
Scans
| Error Message/Issue | Explanation |
|---|---|
| "Scan name already exists" (message on toast) | You tried to create a scan with a name that is already taken by another scan. SOLUTION: Choose another name. |
| "All Data Stores are disabled"(message on toast) | You attempted to run a scan that has all data stores disabled. SOLUTION: Enable at least one data store for the scan. Refer to the Scans for instructions. |
| "The following Data Stores are not accessible: xyz" (message on toast) | You tried to scan a data store that is not accessible. The scan is marked as Failed, and includes a warning icon with the message "The data store xyz included in the scan is not accessible" on mouse-over. SOLUTION: Verify the connectivity from the agent to the data store. Verify the data store configuration. |
| "One or more Data Stores are not accessible." (on mouse-over on the scan fail icon) | The scan failed because the data store that is configured is inaccessible. The data store failed after the scan was launched. SOLUTION: There may be a number of reasons for this. To troubleshoot a failed data store refer to Discovering Sensitive Information. |
| "The following Data Stores have no agent available: xyz" (on mouse-over on the scan fail icon) | You tried to scan a data store that had no agent available when the scan was executed. There is a problem with the agent. The data stores that failed are listed. SOLUTION: This requires additional research, such as checking if the agent is installed on the data store, if it has the right type (local/proxy), if it is of the right OS “flavor” (Linux, Windows), or of the right type (e.g. database). Refer to the Data Discovery and Classification Deployment Guide for more information on troubleshooting this issue. |
| "Data Store has incorrect credentials" (on mouse-over on the scan fail icon) | Data store credentials provided are incorrect so the scan cannot be executed. These data stores are listed. SOLUTION: Update the server credentials for the data store. |
| "One or more Data Stores have incorrect credentials" (message on toast) | The credentials for one or more data stores are no longer valid (credentials modified, user deleted, and so on) preventing the scan from completing. SOLUTION: Reconfigure the data store and re-launch the scan. |
| "The scanner service is not available" (message on toast) | You tried to run a scan with the scan engine unavailable. SOLUTION: Check the status of the scan engine (the CipherTrust Manager server). |
| "The following Data Stores have missing agents: xyz" (message on toast) | This happens when an agent was assigned to the listed data store(s) and then when a scan was launched, for some reason the assigned agent could not be found on the server. SOLUTION: Try to re-assign the agent in the data stores screen. If this does not work, check the agent assigned to the xyz data store. |
| "The following Data Stores have agent errors: xyz" (message on toast) | This happens when a management request for an agent fails (for example, at verification or when setting it as a proxy) during the scan execution. SOLUTION: It is usually a transient issue. Wait a few minutes and run the scan again. If it still fails, check the agent status. |
| "Error processing scan" (message on toast) | This happens when the scan fails in the processing stage, that is when the scan results are being processed by DDC. SOLUTION: Run the scan again. If the error persists, contact Thales Customer Support. |
| "Error connecting to HDFS" (message on toast) | This happens when the scan fails because there is no HDFS connectivity. SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. |
| "Error connecting to PQS" (on mouse-over on a failed scan status) | A scan failed because there is no PQS connectivity. SOLUTION: Check the HDFS configuration in DDC (Hadoop Services) or PQS/Hadoop configuration in your Hadoop deployment. Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-PQS connection. |
| "The Knox logs directory has been filled." | SOLUTION: Change the Knox log level or purge the logs directory. Refer to Changing Knox Log Level in the Thales Data Platform Deployment Guide for details. |
| "Error checking the data allowance" (message on banner) | This happens when DDC is not licensed. DDC sends a request for data allowance to the license server and the server responds that there is no license. SOLUTION: Obtain and install a valid DDC license. Refer to DDC Licensing for licensing information. |
| "One target path is missing" (on mouse-over on the scan fail icon) | A scan failed because one or more target paths are missing. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check which target path is missing (it will be indicated by a yellow exclamation mark in the Targets section). |
| "One database target has incorrect schema" (on mouse-over on the scan fail icon) | A scan failed because one or more database targets have an incorrect schema. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check which target has an incorrect schema (it will be indicated by a yellow exclamation mark in the Targets section). |
| "One database target has incorrect table" (on mouse-over on the scan fail icon) | A scan failed because one or more database targets have an incorrect table. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check which target has an incorrect table (it will be indicated by a yellow exclamation mark in the Targets section). |
| "One target has incorrect file extension" (on mouse-over on the scan fail icon) | A scan failed because one or more targets have an incorrect file extension. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check which target has an incorrect file extension (it will be indicated by a yellow exclamation mark in the Targets section). |
| "One target has nested paths" (on mouse-over on the scan fail icon) | A scan failed because one or more targets have nested paths. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check which target has a nested path (it will be indicated by a yellow exclamation mark in the Targets section). |
| "The target | A file cannot be a target of a data store. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and put a directory as the target path (the failing target path will be indicated by a yellow exclamation mark in the Targets section). |
| "The target | The specified directory used as the target path is invalid. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check the directory. The invalid directory will be indicated by a yellow exclamation mark in the Targets section. |
| "The target | The specified path used ad the target path in inaccessible. SOLUTION: Open the scan for editing, by following the Edit link embedded in the error message, and check the path. The inaccessible path will be indicated by a yellow exclamation mark in the Targets section. |
| "Data Allowance is exhausted" (on mouse-over on the scan status) | A scan was run that exceeded the Data Allowance. All scans running when the Data Allowance gets exhausted (or after that) will fail. From that moment on, you cannot launch new scans. SOLUTION: Contact Thales Customer Support and upgrade your DDC license. Refer to DDC Licensing for licensing information. |
| "Too many sensitive Data Objects found" (on mouse-over on the scan fail icon) | Scan results have a limit on the amount of information (Sensitive Data Object found, infotypes and matches found on each Sensitive Data Object, etc.) and the scan results exceeded this limit. MongoDB data stores are particularly prone to this error. SOLUTION: Split the original scan into multiple smaller scans, re-launch the scans and generate a single report combining the results from all the scans. To reduce the resulting scan size consider scheduling a different scan per Data Store and/or per Classification Profile and/or subpaths (such as folders and tables) in the original scan path. |
| "Error validating Target Paths" (on mouse-over on the scan fail icon) | The target paths validation failed for some reason, usually related to remediation. SOLUTION: Go to the scan view and verify that the target paths exist, and that the guard points exist and are enabled. |
| "There was a fatal error in scan service" (on mouse-over on the scan fail icon) | Unknown error condition on the scan service. Mainly seen when a Database server becomes inaccessible for the scan agent halfway through the scan due to network issues or DB server crash. |
| "Scan RAM exhausted" (on mouse-over on the scan fail icon) | A G-mail scan with a large amount of sensitive data fails with this error. SOLUTION: Increase the Agent's RAM using the ksctl tool. For details refer to Tuning Scan Settings. |
| "Scan results could not be found" (on mouse-over on the scan fail icon) | The scan failed because two independent sub-scans completed in the same second. DDC schedules sub-scans for every path added to the scan. If multiple agents scan a single path, each of them will perform an independent sub-scan. SOLUTION: 1) If the scan failed because of multiple scans finishing in the same second - re-launch the failing scan. 2) If the scan includes multiple agents - reduce the number of agents assigned to Data Stores in the scan. 3) If the scan includes multiple scan paths for a single Data Store - reduce the number of scan paths by scheduling independent scans or by scanning a parent folder that includes multiple scan paths. |
| "Status of one or more Data Stores is failed, go to Data Store list to resolve the issue" (on mouse-over on the failed scan icon) | The agent assigned to one of the data stores used in the scan has failed or could not be found and no alternative agent was found. SOLUTION: Follow the "Data Store list" link embedded in the error message, locate and fix the agent that has failed. Then, go back to the scans page and relaunch the scan. |
| Scan run fails after few minutes with error: "Scan result could not be found" Scan run gets stalled for hours or days | Agent is disconnected. SOLUTION: Identify the agent running the scan and try reconnecting the agent. API request quota limit for the data store is exhausted. SOLUTION: Verify quota usage from cloud provider logs. See API Request Quota Limit to check quota limit for different DS. |
Reports
| Error Message | Explanation |
|---|---|
| "Report name already exists" (message on toast) | You tried to create a report with a name that is already used in another report. SOLUTION: Choose another name. |
| "The version of the scan that was used to generate the report can no longer be found."(message on toast) | The report execution information in TDP references some resource version that is no longer available inside the DDC Database, such as Data Store, Custom Infotype, Classification Profile, or other, probably after restoring DDC from a backup that was taken before the referenced resource was created or modified. SOLUTION: Please restore CipherTrust Manager from a newer backup that includes the referenced resource version, or update the report template to stop referencing the missing resource. |
| "The report template is configured with a scan execution that can no longer be found."(message on toast) | The report template references a concrete scan execution that is no longer found in TDP, probably after restoring TDP from a backup that was taken before the referenced scan execution was completed. SOLUTION: Please restore TDP from a newer backup that includes the information of the referenced scan execution, or edit the report template to reference a valid scan execution date. |
| "The report was generated with version 2.7.0 and is not compatible with new features added in <EXISTING CM VERSION>."(message on toast) | After upgrading from version 2.7.0, Remediated Data Objects are not shown in the reports generated in version 2.7.0. SOLUTION: Regenerate the report to enable the new features. |
Classification Profiles
| Error Message | Explanation |
|---|---|
| "Classification Profile name already exists" (message on toast) | You tried to create a classification profile with a name that is already taken by another classification profile. SOLUTION: Choose another name. |
Licensing
| Error Message | Explanation |
|---|---|
| "DDC License not found - try again in a few minutes if you recently inserted one" (message on banner) | Any action performed in the UI results in this message, because there is no valid DDC license installed. SOLUTION: Obtain and install a valid license. Refer to DDC Licensing for licensing information. |
| "DDC License expired" (message on banner) | Any action performed in the UI results in this message, because your DDC license has expired. SOLUTION: Obtain and install a valid license. Refer to DDC Licensing for licensing information. |
TDP (On-prem)
| Error Message/Issue | Explanation |
|---|---|
| "Hadoop is not active. Please go to DDC Settings -- > Hadoop" (message on toast) | Problem communicating with Hadoop or DDC has not been configured with Hadoop. SOLUTION: Assuming that you have Hadoop deployed in your environment, configure DDC to use it (DDC Settings --> Hadoop in the CipherTrust UI). For a detailed procedure, refer to the Data Discovery and Classification Deployment Guide. |
| "Error connecting to the PQS database" (message on toast) | Problem communicating with the Phoenix Query Server database (i.e. HBase). SOLUTION: Check the PQS configuration in DDC (Hadoop Services) or PQS/Hadoop configuration in your Hadoop deployment. Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-PQS connection. |
| "Error creating the PQS database schema" (message on toast) | Problem communicating with the Phoenix Query Server database. SOLUTION: Check the PQS configuration in DDC (Hadoop Services) or PQS/Hadoop configuration in your Hadoop deployment. Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-PQS connection. |
| "Error using the PQS database schema" (message on toast) | Problem communicating with the Phoenix Query Server database. SOLUTION: Check the PQS configuration in DDC (Hadoop Services) or PQS/Hadoop configuration in your Hadoop deployment. Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-PQS connection. |
| "Error connecting to HDFS" (message on toast) | SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. |
| "Invalid HDFS directory path: Not a directory" (message on toast) | SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. |
| "Incorrect credentials in the HDFS connection" (message on toast) | SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. Check that the authentication service is up and running. |
| "Incorrect HDFS URI" (message on toast) | SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. |
| "Invalid HDFS folder: the path to the folder does not exist" (message on toast) | SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. |
| "Invalid server certificate in the HDFS request" (message on toast) | SOLUTION: Check the HDFS configuration in DDC (Hadoop Services). Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-HDFS connection. |
| "PQS incorrect credentials or authentication service not available" (message on toast) | SOLUTION: Check the PQS configuration in DDC (Hadoop Services) or PQS/Hadoop configuration in your Hadoop deployment. Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-PQS connection. Check that the authentication service is up and running. |
| "Invalid server certificate in the PQS request" (message on toast) | SOLUTION: Check the PQS configuration in DDC (Hadoop Services) or PQS/Hadoop configuration in your Hadoop deployment. Refer to the Data Discovery and Classification Deployment Guide for information on configuring the DDC-PQS connection. |
| "Your system does not meet the 16GB RAM minimum" (message displayed across the top of all DDC screens) | DDC requires at least 16GB of RAM to be able to run properly. SOLUTION: Increase the RAM memory on board to at least the required minumum of 16GB. |
| Knox connection is not configured (message on toast) | Displayed when the user tries to run a scan without setting the Hadoop Knox connection of Hadoop Service. Solution: Configure the Hadoop Knox connection of Hadoop Services under the Access Management > Connection Management tab of CipherTrust Manager. |
| Invalid Knox connection | Displayed when the user tries to run a scan with an invalid Hadoop Knox connection of Hadoop Services. Solution: Verify and reconfigure the Hadoop Knox connection of Hadoop Services under the Access Management > Connection Management tab of CipherTrust Manager. |
| HDFS settings don't exist | Displayed when the user tries to run a scan without configuring the HDFS settings of Hadoop Services. Solution: Configure the HDFS settings under Settings > Hadoop Services in DDC. |
| Livy settings don't exist | Displayed when the user tries to run a scan without configuring the Livy settings of Hadoop Services. Solution: Configure the Livy settings under Settings > Hadoop Services Settings in DDC. |
| --Pending-- (message on toast) | Displayed when a user tries to save the HDFS settings but the hostname or port from the connection created in Connections Management are wrong or not reachable. |
| Livy connection refused (message on toast) | Displayed when a user tries to save the Livy settings but the hostname or port from the connection created in Connections Management are wrong or not reachable. |
| Invalid knox server certificate (message on toast) | Displayed when a user tries to save the HDFS or Livy settings but the certificate is invalid. |
| Invalid HDFS URI path (message on toast) | Wrong parameter added in the DDC Hadoop settings. |
| Invalid HDFS folder (message on toast) | Wrong parameter added in the DDC Hadoop settings. |
| Invalid Livy URI path (message on toast) | Wrong parameter added in the DDC Hadoop settings. |
| 503 Service Unavailable (message when configuring HDFS/Livy) | If you are using the CipherTrust Manager's proxy feature, make sure it is properly configured. It should be possible to directly access TDP, to resolve its name, and to allow HTTPS connections to port 8443. Check your proxy software manual to get those rules configured. |
| Connection error "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)." (message in ambari-agent.log) Server lost heartbeat from all nodes (message on Ambari UI) | Ambari CA certificate is corrupted or has expired. Solution: See Expired/Corrupted Ambari CA Certificates to resolve this issue. |
| TDP silent installation failure | Occasionally, TDP silent installation may fail when setting up node cluster. Solution: See TDP Silent Installation Failed to resolve this issue. |
Remediation
| Error Message | Explanation |
|---|---|
| Remediation could not be done (on mouse-over on the scan fail icon) | The scan with remediation failed. Please check the remediation configurations before starting it again. |
TDPaaS
The below sections list various error messages that you may encounter when working with TDPaaS, with their explanations and possible solutions.
Scans & Reports
| Error Message | Explanation |
|---|---|
No valid Hadoop Knox or Data Management Services connection exist | Displayed when the user tries to run a scan with an invalid Hadoop Knox connection of Hadoop Services. Solution: Verify and reconfigure the Hadoop Knox connection of Hadoop Services under the Access Management > Connection Management tab of CipherTrust Manager. |
None of Hadoop Knox & Data Management Services connection exist | Displayed when the user tries to run a scan without configuring either Hadoop Knox or Data Management Services connection. Solution: Configure the Hadoop Knox connection of Hadoop Services under the Access Management > Connection Management tab of CipherTrust Manager or Provision Data Management Services settings under Settings > Data Management Services in DDC. |
The Data Management Service settings don't exist | Displayed when the user tries to run a scan without configuring the Data Management Services settings. Solution: Configure Data Management Services under Settings > Data Management Services in DDC. |
None of Data Management Services & Hadoop Services settings exist | Displayed when the user tries to run a scan without configuring the HDFS & Livy settings of Hadoop Services and Data Management Services. Solution: Configure the HDFS & Livy settings under Settings > Hadoop Services in DDC or Provision Data Management Services under Settings > Data Management Services in DDC. |
Connection is Lost: Data Management Service connection is not responding. | Displays when the connection to the Data Management Service is interrupted. Soluton: Ensure Data Management Services are correctly configured in the Settings tab. |
Data Management Services Settings & Provisioning
| Error Message | Explanation |
|---|---|
Error: Hdfs Settings can't be changed | Data platform was switched from Data Management Services to Hadoop Services after a successful scan execution. Solution: This action is not allowed. After executing a scan with Data Management Services, you can't switch to Hadoop Services. |
Error: Livy Settings can't be changed | Data platform was switched from Data Management Services to Hadoop Services after a successful scan execution. Solution: This action is not allowed. After executing a scan with Data Management Services, you can't switch to Hadoop Services. |
Error: Tdpaas Settings can't be changed | Data platform was switched from Hadoop Services to Data Management Services after a successful scan execution. Solution: This action is not allowed. After executing a scan with Hadoop Services, you can't switch to Data Management Services. |
TDPaaS Provision Data can't have empty fields | Data Management Services provisioning returned empty fields in response. Solution: Retry Data Management Services provisioning. |
Error: region param is invalid | Data Management Services was provisioned with an invalid or unsupported region. Solution: Try provisioning with another supported region. |
Error: missing field %s | Data Management Services configuration was attempted with a missing value for a mandatory field. Solution: Specify the missing field value and reconfigure the service. |
Error: Data Management Service Settings could not get provisioned | Data Management Services provisioning failed to create and fetch the Data Management Services settings. Solution: Retry Data Management Services provisioning. |
Error: Operation forbidden, cannot change settings - scan is running | Switched the data platform from Hadoop Services to Data Management Services or from Data Management Services to Hadoop Services while scan execution was in progress. Solution: Switch the data platform after scan execution gets failed. After a scan is successfully completed, you can't switch the data platform type. |
Error: Operation forbidden, cannot change settings - TDPaaS settings is fixed | Switched the data platform from Hadoop Services to Data Management Services after a successful scan execution. Solution: This action is not allowed. After a successful scan execution with Data Management Services, you can't switch the data platform. |
Error: Operation forbidden, cannot change settings - TDP-OnPrem settings is fixed | Switched the data platform from Data Management Services to Hadoop Services after a successful scan execution. Solution: This action is not allowed. After a successful scan execution with Hadoop Services, you can't switch to Data Management Services. |
Other Errors
| Error Message | Explanation |
|---|---|
DDCTDPaaSDataprocLaunchJobOperation error | Error relates to starting the scan job while using Data Management Services. Solution: This is an unexpected error. Details about the specific cause of error are logged in the Loki Audit records, for each failed attempt of starting the scan job. Please contact customer support to resolve this issue. |
DDCTDPaaSDataprocGetJobStatusOperation error | Error in performing the get scan job status while using Data Management Services. Solution: This is an unexpected error. Details about the specific cause of error are logged in the Loki Audit records, for each failed attempt of getting the scan job. Please contact customer support to resolve this issue. |
Error: Both LivySettings & Data Management Service are set | Livy Settings of the Hadoop Services and Data Management Services settings are set concurrently. Solution: This is an unexpected error. Please contact customer support to resolve this issue. |
Error: Neither Livy nor Data Management Service is set | Neither Livy Settings of Hadoop Services nor Data Management Services settings are set. Solution: Ensure Hadoop Services or Data Management Service is correctly configured on the DDC Settings tab. |
Error: cannot initialize StorageConnector, Hdfs & Data Management Service are set | Both HDFS Settings of the Hadoop Services and Data Management Services settings are set concurrently. Solution: This is an unexpected error. Please contact customer support to resolve this issue. |
Error: Neither HDFS nor Data Management Service is set | Neither HDFS Settings of Hadoop Services nor Data Management Services settings are set. Solution: Ensure Hadoop Services or Data Management Service is correctly configured on the DDC Settings tab. |
Info Log Messages
This section lists the various information messages that Data Discovery and Classification sends to CipherTrust Manager log.
OLEANDER INFO
| Error Message | Explanation |
|---|---|
| "[Background-Processes] This node was NOT selected as active node. Turning off background processes" | Related to clustering, self-explanatory error. |
| "[Background-Processes] This node was selected as active node. Turning on background processes" | Related to clustering, self-explanatory error. |
| "[Background-Processes] Recovering collectors" | Oleander has received a valid license and recovers the collector processes that were stopped. |
| "[Background-Processes] Updating license from DMV" | Oleander is requesting from DMV the licenses available for DDC. |
| "[Background-Processes] Global license status set to nil" | Oleander is unlicensed. |
| "[Background-Processes] Global license status set to ", "newLicenseStatus", *newLicenseStatus | Oleander license status is whatever newLicenseStatus is. |
| "[Background-Processes] Checking HDFS connectivity | Oleander is sending a ping to HDFS to check the connectivity. |
| "[Background-Processes] Cannot connect with HDFS", "error", err | Oleander's ping against HDFS has failed. Oleander has no connectivity with HDFS. |
| "[Background-Processes] HDFS connectivity successful" | Oleander has successfully performed a ping against HDFS. Oleander has connectivity with HDFS. |
| "[Background-Processes] Checking PQS connectivity" | Oleander is sending a ping to PQS to check the connectivity. |
| "[Background-Processes] Cannot connect with PQS", "error", err | Oleander's ping against PQS has failed. Oleander has no connectivity with PQS. |
| "[Background-Processes] PQS connectivity successful" | Oleander has successfully performed a ping against PQS. Oleander has connectivity with PQS. |
| "[Background-Processes] License status has changed" | Oleander license status has changed since the last license cron execution. |
| "[Background-Processes] Initializing fast cron: Oleander unlicensed","UnlicensedCronFrequency", o.Config.UnlicensedCronFrequency | Oleander is unlicensed so the license cron (which asks DMV for a DDC license) increments its frequency. The frequency is defined in the docker-compose file. |
| "[Background-Processes] Initializing fast cron: waiting for Hadoop connectivity","HadoopConnectivityCronFrequency", o.Config.HadoopConnectivityCronFrequency | Oleander has a valid license but does not have connectivity with Hadoop. |
| "[Background-Processes] Initializing slow cron: Oleander licensed and Hadoop connectivity successful", "RunningCronFrequency", o.Config.RunningCronFrequency | Oleander has a valid license and has connectivity with Hadoop, the license cron decrements its frequency. The frequency is defined in the docker-compose file. |
| "[Background-Processes] About to run NO LICENSE scenario" | Oleander has no valid license so it will stop all running scan watchers, automatic agent selections and scan schedules. |
| "[Background-Processes] About to run VALID LICENSE scenario" | Oleander didn't have a valid license but now it has one so all scan watchers, automatic agent selections and scan schedules that were stopped are being resumed. |
| "[Background-Processes] Recovering automatic agent selection" | Part of the VALID LICENSE scenario above. |
| "[Background-Processes] Starting automatic agent selection for pending datastores", "pending datastores", strings.Join(ds, ", ") | Part of the VALID LICENSE scenario above. |
| "[Background-Processes] Recovering watchers" | Part of the VALID LICENSE scenario above. |
| "[Background-Processes] Starting watchers for ongoing scans", "ongoing scans", strings.Join(ss, ", ") | Part of the VALID LICENSE scenario above. |
| "[Background-Processes] Recovering scan schedules" | Part of the VALID LICENSE scenario above. |
| "[Background-Processes] Migrating PQS database" | Self-explanatory message. |
| "[Background-Processes] Deleting background process for scan", "name", sc.Scan.Name | Oleander has no valid license so all scans are being stopped as well as their corresponding background processes. |
| "[Background-Processes] Updating status to FAILED for scan", "name", sc.Scan.Name | Oleander has no valid license so all scans are being stopped (with status FAILED). |
| "Cannot connect to HDFS", "settings", hdfsSettings, "error", err | Oleander's ping against HDFS has failed. Oleander has no connectivity with HDFS. |
| "Cannot connect to PQS", "Settings", pqsSettings, "error", err | Oleander's ping against PQS has failed. Oleander has no connectivity with PQS. |
| "[Datastores] Agent selected: ", "Name", a.Name | Oleander has found and assigned a suitable agent for the mentioned data store. |
| "Unable to connect with datastore", "datastore name", d.Name, "error", err | A probe against the mentioned data store has failed during a test connectivity check. |
| "Instantiating new scrim helper" | Oleander is instantiating a new ScrimHelper object which is used for communication with Scrim, Minerva, Sallyport and DMV. |
| "Instantiating new hdfs scan collector" | Oleander is instantiating a new HDFSCollector object which is used for communication and processing with HDFS. |
| "Instantiating scheduler cron" | Oleander is instantiating a new SchedulerCron and starting the background processes. |
| "[WARNING] PQS connector can not be closed in GetSummaryReport service" | Oleander was unable to close the PQS connector while executing a GetSummaryReport operation. |
| "[WARNING] PQS connector can not be closed in GetDatastoresDetailsReport service" | Oleander was unable to close the PQS connector while executing a GetDatastoresDetailsReport operation. |
| "[WARNING] PQS connector can not be closed in GetReportTemplate service" | Oleander was unable to close the PQS connector while executing a GetReportTemplate operation. |
| "[Scan-Actions] [WARNING] PQS connector can not be closed in GetScanExecutions service" | Oleander was unable to close the PQS connector while executing a GetScanExecutions operation. |
| "[Scan-Watchers] Stop watcher signal received", "scan", sc.Scan.Name | A running scan watcher has received a STOP signal and the current execution is cancelled. |
| "[Scan-Watchers] Watcher has failed stopping scan from scanned service" | The scan has been in an INTERRUPTED status for too long so Oleander has tried to stop the scan in ER2 but the request was unsuccessful. |
| "[Scan-Watchers] Watcher has detected different status", "scan", sc.Scan.Name, "current status", sc.Scan.ScanProcess.Status, "scan service status", er2MappedStatus | The scan watcher has detected a change in the scan status, scan status will be updated in DDC. |
| "[Scan-Watchers] Interrupted status received", "MaxTimeInterruptedState", o.Config.Er2MaxInterruptedTime, "scan", sc.Scan.Name, "current status", sc.Scan.ScanProcess.Status, "scan service Mapped status", er2MappedStatus, "scan service status", er2Status, "InterruptedTimestamp", sc.InterruptedTimestamp | The scan watcher has received an INTERRUPTED status for the scan, the scan watcher will continue asking until the status changes or the timeout is exceeded. |
| "[Scan-Watchers] Scan has finished... collector starting", "scan", sc.Scan.Name | The scan watcher has received a COMPLETED status for the scan, the scan status is set to PROCESSING in DDC and the scan collector starts. |
| "DDCScanActionComplete" | Scan audit record, logged after a successful scan execution. |
| "DDCScanActionFail" | Scan audit record, logged after a failed scan execution. |
| "AggregatedReportOperationCompleteDDC" | Report audit record, logged after a successful aggregated report generation. |
| "ScanTrendReportOperationCompleteDDC" | Report audit record, logged after a successful scan trend report generation. |
SUNDEW INFO
| Error Message | Explanation |
|---|---|
| "Asking for license status" | Sundew is requesting from Oleander the DDC license status. |
| "Active license information received", "http ret code", c | Sundew has received a valid license status from Oleander. |
| "Starting scan service..." | Sundew is launching ER2. |
| "Connectivity test successful, scan service is up" | Sundew confirms that ER2 is up-and-running. |
| "Generating scan service license...", "ID", s.er2Status.lastIssuedLicenseID | Sundew is generating a license for ER2. |
| "Injecting license into scan service" | Sundew is injecting the generated license into ER2. |
| "License won't be generated/refreshed in this iteration", "ID", s.er2Status.lastIssuedLicenseID, "checks (intervals)", s.Config.GeneratingLicenseIterval-s.er2Status.lastIssuedLicense | No action is required from Sundew for this iteration regarding licensing. |
| "Not active license or product not licensed response received", "http ret code", c | Sundew has received an invalid license status from Oleander. |
| "Scan service stopped" | Sundew is stopping ER2. |
| "Unexpected error code in response for /license/status", "http ret code", c | Sundew asked Oleander for the license status but got an unexpected response. |
Error Log Messages
This section lists the various error messages that Data Discovery and Classification sends to CipherTrust Manager log.
OLEANDER ERROR
| Error Message | Comment/Explanation |
|---|---|
| Error connecting to the scan service | This error means a connectivity issue between the oleander and the sundew/ER2. |
| DDC Error creating the database schema.: error executing http request. Code: 500 | To check the connectivity with Hadoop external data base a schema is created. So this means that there is no connectivity with Hadoop. (also, possibly related to "Error creating the PQS database schema" in the UI) |
| "CLIENT_CREDENTIAL_PARTITION is not set" | CLIENT_CREDENTIAL_PARTITION variable is not set in the config object. |
| "[Background-Processes] Error retrieving license from DMV", "error", err | Oleander GetLicenses request against DMV has failed. This error could have been caused by DMV being down. |
| "[Background-Processes] Error killing all agents selections" | When the oleander instance loses its license or the current license expires, all ongoing agent selections will stop. This error is caused by an internal Oleander issue while this agent selections are being shut down. |
| "[Background-Processes] Error killing all scan watchers" | When the oleander instance loses its license or the current license expires, all ongoing scan tracking will stop. This error is caused by an internal Oleander issue during this scan tracking shut down. |
| "[Background-Processes] Error removing all scan schedules due to DDC not licensed" | When the oleander instance loses its license or the current license expires, all scheduled scans will be stopped. This error is caused by an internal Oleander issue during this scheduled scans stopping. |
| "[Background-Processes] Error starting agent selection for datastores" | The listed Data Stores have no agent available. |
| "[Background-Processes] Error starting scans watchers" | When the oleander instance receives a valid license from DMV, all stopped agent selections must be resumed. This error is caused by an internal Oleander issue during this agent selections resuming. |
| "[Background-Processes] Error starting cron schedules" | When the oleander instance receives a valid license from DMV, all stopped scheduled scans must be resumed. This error is caused by an internal Oleander issue during this scheduled scans resuming. |
| "[Background-Processes] Error trying to migrate the PQS database" | During the PQS configuration, an error occurred trying to apply the changesets to update the Database, create the tables, etc... |
| "[Background-Processes] Error creating cron", "error", err | Background_processes service constantly creates crons for license status checking against DMV, this error is caused by an internal Oleander issue during the creation of one of this crons. |
| "[Background-Processes] Error with unmarshal." | Any "unmarshal/unmarshalling" error is caused by an internal Oleander issue converting a golang object to JSON format or vice versa. |
| "[Background-Processes] Error while trying to update status to FAILED for datastore.", "name", ds.Name, "error", err | When background_process services gets an invalid or expired license from DMV all running scans must be stopped and set as FAILED, this message indicates an internal Oleander error changing the scan status for some scan. |
| "[Background-Processes] Error while trying to retrieve scans from background processes table" | When the oleander instance receives a valid license from DMV, all stopped normal scans must be resumed. This error is caused by an internal Oleander issue accessing the background processes table, which contains all the information for the scans resuming. |
| "[Background-Processes] Watcher has failed updating scan status for scan", "name", sc.Scan.Name, "error", err | background_processes service has the responsibility to track the running scans and update their Oleander status, this error message indicates an internal Oleander issue updating the scan_process table. |
| "Cannot retrieve the HDFS settings", "error", err | Connectivity or internal Oleander error trying to retrieve HDFS settings from Citrus. |
| "Cannot retrieve the PQS settings", "error", err | Connectivity or internal Oleander error trying to retrieve PQS settings from Citrus. |
| "Cannot find the country", "error", err | When a user creates a Location and sets a country that is not registered in our DB. |
| "Cannot find the state", "error", err | When a user creates a Location and sets a state ID that is not registered in our DB. |
| "Trying to verify if the country has states", "error", err | When a user creates a Location and he sets a state "Name" that is not registered in our DB. |
| "Missing tag %s for the default classification profiles", r | When inserting default classification profiles, there is no tag that matches the correct Regulation |
| "[Datastores] Error encrypting connection for datastore: ", "Name", d.Name | Error when calling Scrim Helper for the encryption of the Connection field for a data store. |
| "[Datastores] Error while trying to create background process resource for datastore: ", "Name", d.Name | When a datastore is created, a new row is inserted into the Background Processes table for further tracking, this error message indicates an internal Oleander issue inserting that row. |
| "[Datastores] Error selecting agent for datastore ", "Name", dsAAS.Name, "error", err | No suitable agent has been found for this data store. |
| "[Datastores] Error while trying to retrieve datastore for background process, agent selection might fail: ", "error", err | Error retrieving a data store from DB for background processes purposes in a data store.update operation. |
| "[Datastores] Error running automatic agent selection", "error", errAgentUpdate | An internal Oleander error has occurred while trying to update the status of a data store during the automatic agent selection. |
| "[Datastores] Error while trying to unmarshal datastore from background processes" | An internal Oleander error has occurred while trying to translate a JSON object to a golang object while recovering the automatic agent selections. |
| "Error closing the json file", "error", err | Oleander failed trying to close a JSON file. |
| "Error trying to close the families json file" | Oleander reads a families JSON file for tartup DB population. This error message indicates an internal Oleander error while closing this file. |
| "Error initializing the account", "error", *errPtr | Oleander failed trying to set the initialization status to the accounts map. |
| "Error closing oleander", "error", err | Oleander service could not be closed. |
| "Error starting background processes", "error", err | Some error has occurred trying to execute all background processes. |
| "Error connecting to HDFS", "error", err | This happens when the scan fails because there is no HDFS connectivity. |
| "Error trying to close the info types json file" | Oleander reads an infotypes JSON file for startup DB population. This error message indicates an internal Oleander error while closing this file. |
| "Error connecting to PQS", "error", err | This happens when the scan fails because there is no PQS connectivity. |
| "Error closing the temporary file", "FileName", f.Name(), "error", err | Oleander failed closing the temporary file used for decrypting the raw data file. |
| "Error deleting the temporary file", "FileName", f.Name(), "error", err | Oleander failed deleting the temporary file used for decrypting the raw data file. |
| "Error trying to close the zip file", "error", merry.Wrap(err).WithHTTPCode(http.StatusBadRequest) | Oleander failed closing the zip file used for decrypting the raw data file. |
| "Error changing the scan status", "error", err | Internal Oleander error while trying to update the scan status in the Scan Process table. |
| "[Scan-Launcher] Error while trying to create background process resource for datastore", "name", rsc.Name | Oleander inserts a row in the Background Processes table for further scan tracking - this message indicates an internal Oleander error while performing this insert. |
| "[Scan-Launcher] The scan watcher returned an error", "error", err | Generic error message for any issue during the scan watcher process. |
| "[Scan-Launcher] Agent not found for datastore", "DS name", s.ScanDatastores[i].Datastore.Name, "error", err | The listed Data Stores have no agent available. |
| "[Scan-Launcher] Error getting absolute paths", "DS name", s.ScanDatastores[i].Datastore.Name, "error", err | Oleander was unable to retrieve the absolute paths of the mentioned data store for further scan execution. |
| "[Scan-Launcher] Error getting connection path", "DS name", s.ScanDatastores[i].Datastore.Name, "error", err | Oleander was unable to retrieve the connection paths of the mentioned data store for further scan execution. |
| "[Scan-Launcher] Error retrieving the oleander context" | Error while generating the Oleander service user. (this is the context for executing actions on behalf of the itself, instead of a specific user). |
| "[Scan-Launcher] Error while validating the scan" | This is a generic error, one of the steps of the validation has failed. (could be that some data stores are not ready, a probe has failed, etc) |
| "[Scan-Launcher] Error while retrieving extensions" | The extensions necessary for the scan execution could not be retrieved from the DB. |
| "[Scan-Launcher] Unable to initialize background process" | The background process object could not be initialized so the scan watcher cannot start. |
| "[Scan-Actions] Error found while trying to delete background process resource" | When Oleander has finished tracking a scan it removes the corresponding row from background processes table - this error message indicates an internal Oleander issue removing that row. |
| "[Scan-Actions] Error trying to execute the Scan scheduled run", "error", err | Oleander failed trying to add a scan schedule for a scan. |
| "Error changing the scan status", "error", err | Generic error message for an issue during the update of the scan status. |
| "[Scan-Watchers] Watcher has failed getting scan status from scanned service", "scan", sc.Scan.Name, "error", err | This indicates an underlying connectivity issue with Sundew. |
| "[Scan-Watchers] Watcher has failed updating scan status", "scan", sc.Scan.Name, "error", err | Error updating the scan status in the scan process table. |
| "[Scan-Watchers] Error processing the report: ", "error", err | Oleander scan collector has failed trying to create the scan collector background process for the mentioned scan. |
| "[Scan-Watchers] Scan aborted, maximum wait for scan results exceeded", "MaxTimeInterruptedState(minutes)", o.Config.Er2MaxInterruptedTime/60, "scan", sc.Scan.Name, "current status", sc.Scan.ScanProcess.Status, "scan service status", er2MappedStatus | The scan has been in INTERRUPTED er2 state for too long, when it exceeds the timeout the status is changed to FAILED in DDC. |
| "Error trying to parse er2 polling frequency from env" | Oleander failed trying to read the ER2 POLLING FREQUENCY env variable from the config object. |
| "Error while trying to retrieve scans from background processes table", "error", err | Oleander tried to retrieve all SCAN rows in the background processes table but it failed. |
| "Error trying to recover the scan watchers", "error", err | One of the scan watchers instantiated by the recovery system has returned an error and stopped working. |
| "Wrong datastore credentials" | Oleander was unable to reach a data store due to credentials failure. |
| "Wrong target path defined" | A scan has failed because the target path defined was not valid. |
| "Wrong db schema in target path" | A scan has failed because the db schema defined was not valid. |
| "Wrong db table in target path" | A scan has failed because the DB defined was not valid. |
| "Wrong defined file extension in target path" | A scan has failed because a file was specified without an extension. |
| "Error connecting to the scan service" | Oleander is unable to connect with ER2. |
| "Error processing the scan reports" | Generic error while trying to process a scan report. |
| "No Data Allowance Licensing detected" | Oleander does not have a Data Allowance record. |
| "Error reading the scan report" | Generic error while trying to read a scan report. |
| "Probing path can be launched only on folders. Files are not supported" | Scan failed because a file was specified as target instead of a folder. |
| "Probing a File or Directory that does not exist" | Scan failed because the specified directory does not exist. |
SUNDEW ERROR
| Error Message | Explanation |
|---|---|
| "CLIENT_CREDENTIAL_PARTITION is not set." | CLIENT_CREDENTIAL_PARTITION variable is not set in the config object. |
| "Error trying to ask for license status", "error", err | Sundew failed trying to request DDC status from Oleander. |
| "Error trying to start the scan service", "error", err | Sundew failed trying to start ER2. |
| "Connectivity test to the scan service failed", "error", conErr | Sundew failed trying to ping ER2. ER2 is down. |
| "Error during validation of received license", "error", err | Sundew failed trying to validate the DDC license retrieved from Oleander. |
| "Error parsing the license ID", "error", err | Sundew failed trying to parse the DDC license ID received from Oleander. |
| "Error while injecting license inside scan service", "error", err | Sundew failed trying to inject the generated license into ER2. |
| "Error stopping scan service", "error", err | Sundew failed trying to stop ER2. |
| "Error closing the response body", "error", err | Sundew failed trying to close the HTTP response body from a request. |
| "Error closing sundew", "error", err | Error when trying to close the Sundew service. |
Checking the System Health
If you run into a problem with DDC, use this procedure to check the system health before you report anything to Thales Customer Support.
(if CM is clustered) Check there is a defined DDC active node and check if it is part of the cluster.
Refer to the information in Identifying the Active DDC Node for details.
Check the available disk space using Linux console or disk-usage API. If DDC node free disk space below 20% the scan service will not be available.
Use the GET disk-usage API as descried in the CipherTrust Manager API Guide.
API: /v1/ddc/server-statistics/disk-usage
Description: Returns disk usage statics of scanner service.
Sample response
{ "diskUsage": 82.23, "degradedMode": false, "error": "" }Access the Linux console (of the DDC active node in case of a CM cluster) as the user ksadmin.
Run the
df -hcommand and verify that the "Use%" value for the "/" partition is under 80%.ksadmin@ciphertrust:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 3.7M 1.6G 1% /run /dev/sda2 49G 12G 35G 26% / tmpfs 7.9G 0 7.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup /dev/sda1 464M 130M 307M 30% /boot tmpfs 1.6G 0 1.6G 0% /run/user/0
Check the status of the DDC License.
Refer to the information in Viewing the License Status for details.
Check the configuration and status of the TDP connection.
Refer to the information in Configuring TDP for details.
Check TDP health status.
Refer to the Thales Data Platform documentation for details.
Check the Kylo logs and look for messages related to Oleander and Sundew.
Access to the linux console of the DDC active node as the ksadmin user.
To see the DDC logs in real time, run the following command:
tail -f /opt/keysecure/logs/keysecure.system.log | egrep "\| (oleander|sundew)"To copy all the ddc logs into a single file, run:
> /tmp/ddc.log ls -r /opt/keysecure/logs/keysecure.system.log* | while read FILE; do ext="${FILE##*.}"; if [ "$ext" = "gz" ]; then CMD="zcat"; else CMD="cat"; fi; $CMD $FILE | egrep "\| (oleander|sundew)" >> /tmp/ddc.log; doneTo copy the ddc logs filtered by a specific date (YYYY-MM-DD), run:
> /tmp/ddc-YYYY-MM-DD.log ls -r /opt/keysecure/logs/keysecure.system.log* | while read FILE; do ext="${FILE##*.}"; if [ "$ext" = "gz" ]; then CMD="zcat"; else CMD="cat"; fi; $CMD $FILE | | egrep "^YYYY-MM-DD" | egrep "\| (oleander|sundew)" >> /tmp/ddc-YYYY-MM-DD.log; doneTip
Replace the YYYY-MM-DD bit with the date that you want to filter by.
Agent connectivity and clock.
All agents
Go to the Agents page, by clicking the Data Discovery and Classification panel in the CipherTrust Manager dashboard, then click to expand the Data Stores menu in the navigation panel on the left, then click Agents.
Check whether agents are listed there with right status.
Check if agent clocks are ok.
For more details please refer to information in Agents.
Specific Agent
Verify the logs generated by the DDC agent.
In Linux:
/var/lib/er2/agent.logIn Windows:
C:\Program Files (x86)\Ground Labs\Enterprise Recon 2\agent.log
Verify the connection from the agent to the CM server.
- In Linux:
Run the
er2-config -tcommand:root@agent1:~# er2-config -t Master server is not specified, using the default master server: sundew Testing connection setting... Test SUCCESS. Saving settings Target initial group is not specified, using the default public key [preinstalled] Configuration updated, please restart agent service The configuration has been saved. Please restart the agent for the changes to take effect.In Windows:
Open the Configure Agent application.
Ensure it has the correct server IP or hostname (DDC active node in case of a CM cluster).
Click the Test Connection button.
Verify the disk space.
Resolving High Disk Usage Issues
DDC performs automatic disk clean up in the backend when a scan run is complete or stopped. However, this process is skipped for a failed scan and all other subsequent scans, whether they fail or succeed. As a result, disk consumption may increase over time, potentially exceeding threshold levels.
Tip
Use the Linux console or disk-usage API to regularly monitor disk usage, and take proactive measures to prevent disk consumption from exceeding threshold levels. See step 2 under Checking the System Health for details.
DDC continuously monitors disk usage and displays alert banners when disk consumption reaches a certain threshold value.
Warning Banner
DDC displays a warning banner when disk usage reaches 60% or above.
When you see a warning banner, perform the following tasks to manage disk space:
Start the disk clean up process via warning banner.
Click Start Clean Up Process.
In the consent dialog, click Clean Up.
This action will stop all running scans and initiate the clean-up process. After the clean up process is complete, you can rerun the scans manually.
Alternatively, click Cancel and allow the scan runs to complete or stop them manually, before restarting the clean-up process.
Note
After performing cleanup, it takes some time for DDC to synchronize and reflect the actual disk space, and stop displaying the warning banner.
Delete backup files to reduce disk consumption.
Increase the disk space.
Degraded Banner
DDC enters the degraded mode when disk usage reaches 85% or above and displays a degraded banner with an option to initiate clean up process. DDC exits the degraded mode when the disk space is reduced to 70% or below.
Note
When degraded mode is reached, DDC becomes inoperable and all the APIs related to Scans, Datastores, Agents, Classification Profiles, Infotypes, and Report will not function.
When you see a degraded banner, perform the following tasks to manage disk space:
Start the disk clean up process via degraded banner.
Click Start Clean Up Process.
In the consent dialog, click Clean Up.
During the clean up process:
All the running, paused, and auto-paused scans will fail.
No trace logs will be available for the failed scans.
Note
After performing cleanup, it takes some time for DDC to synchronize and reflect the actual disk space, and stop displaying the degraded banner.
Delete backup files to reduce disk consumption.
Increase the disk space.
Note
When disk cleanup is performed by increasing the disk space and/or deleting backup files, all running scans will continue to proceed uninterrupted. Additionally, if a scan was paused, it can be manually resumed.
