Records

Audit records are logs of auditable events viewable in CipherTrust Manager in a user-friendly JSON format.

Server records contain operations performed by the CipherTrust Manager. You can also view the audit records that are uploaded from the CipherTrust Manager Clients (for example, CTE). Consult documentation for a specific connector for information on configuring and interpreting client records. The Logs chapter of CTE Agent for Linux Advanced Configuration and Integration Guide contains details on CTE client record format and effects codes.

Audit records can be locally fetched and searched using the CipherTrust Manager REST API, CLI, or the GUI. The records available through these queries are scoped to the domain permissions of the logged in user.

Both server and client records can be configured to raise alarms.

There are three options to store and display records:

• Loki Grafana Microservice

• Log Forwarding

• Local Database Logging

Loki Grafana Microservice Option

Records for the local node are automatically logged to an internal Loki Grafana log aggregation microservice, where they can be viewed and queried. This option has the best performance locally. In the GUI, Loki records are visible under Records > Loki Audit Records. The retention period for records is configurable, with a default value of 240 hours. These records are not replicated in cluster.

Log Forwarding Options

Records can also be sent to an external logging system through a log forwarder. Supported log forwarder types are Elasticsearch, Loki, and Syslog server. These options allow you to view records for a cluster in aggregate.

Local Database Logging Option

To view database audit records for CipherTrust Manager activity, you must first enable local database logging. Then, in the GUI, go to Records > Server Records. To view client records in the GUI, go to Records > Client Records. The Server Audit Records database table is automatically trimmed when it reaches 10 million records in size.

Server Records Parameters

Loki Grafana Microservice

You can list and query records on the Loki Grafana Microservice, and download the records to a file. As well, you can configure the retention period, after which records are deleted.

Listing and Querying Records With Loki Grafana Microservice

You can query both server audit records and client audit records from the local CipherTrust Manager node to return records. The available records depend on the configured retention period after which records are deleted. The default retention period is 240 hours.

View and Query Records in the GUI

Application administrators, such as the admin user can view and query records. There is a basic query and an advanced query to limit which Loki audit records are displayed.

To view records, navigate to Records>Loki Audit Records. Select Server Records or Client Records.

Basic Query

Select the filter icon () to filter on particular fields or within a specified time range. The following parameters are available to filter on:

• Severity

• Status

• By

• Source

• Client IP

• Message

• Max Records

• Date and Time

• Start Date

• End Date

Select Apply Query to view which records are returned.

Advanced Query

Select Advanced Query to input a query following Loki Grafana's LogQL syntax. If you have applied a basic query, it is already inputted in LogQL syntax. There are options to save and load queries from the Advanced Query menu.

Select Apply Query to view which records are returned.

List and Query Records with ksctl

Application administrators, such as the admin user can list records in the Loki Grafana microservice with ksctl loki query_range --server-audit-records and ksctl loki query_range --client-audit-records.

The query parameters follow Loki Grafana's LogQL syntax for doing a query over a specified time range. Use ksctl loki query_range commands to input parameters for limit, start, end, step, interval, and direction.

Save Queries With Loki Grafana Microservice

The CipherTrust Manager web console UI allows you to save and load queries for convenience.

We also provide two public server records queries available to load for Google CSE Records and Google Cloud EKM Records, which are CipherTrust Cloud Key Manager (CCKM) integrations.

To save a query:

1. Login as an Application administrator, such as the admin user.

2. Navigate to Records>Loki Audit Records. Select Server Records or Client Records.

3. Select Advanced Query.

   A text box appears.

4. Input a query following Loki Grafana's LogQL syntax.

5. Optionally, select Apply Query to view which records are returned.

6. Select Save Query.

   The Save Query popup window appears.

7. Provide a Name and review the Query Preview.

   Note

   You can only save the query as Private.

8. Select Save to finalize.

To load a query:

1. Login as an Application administrator, such as the admin user.

2. Navigate to Records>Loki Audit Records. Select Server Records or Client Records.

3. Select Advanced Query.

   A text box appears for query input.

4. Select Load Saved Query.

   The Load Saved Query popup appears.

5. Select one of the saved queries on the left.

   Two public queries for CCKM integrations, Google CSE Records and Google Cloud EKM Records, are available, as well as any previously saved private queries.

6. Select Load.

   The query is loaded and applied.

Download Loki Audit Records to a File

You can download the Loki Audit Records displayed in the CipherTrust Manager web console UI to a file.

To download the records to a file:

1. Login as an Application administrator, such as the admin user.

2. Navigate to Records>Loki Audit Records. Select Server Records or Client Records.

3. Adjust the query as desired, to limit which Loki audit records will be downloaded.

4. Click Download Logs in the upper right corner.

   The logs are downloaded to a file loki_audit_log.txt.

Configuring Retention Period for Loki Grafana Microservice

By default, all records generated since the CipherTrust Manager is deployed are retained in Loki. To preserve disk space, users in the 'admin' group can set a retention period for which Loki logs are retained.

The retention period has the following characteristics:

• The minimum value is 24 hours.

• A valid value must be a multiple of 24 hours.

• The retention period must be a string that can be parsed using Go library’s time.Duration type, such as 24h.

To set the Loki configuration retention period:

ksctl loki config modify --retention-time "<desired_retention_time>"

To view the Loki configuration retention period:

ksctl loki config get

Configuring remote Syslog server

Records can optionally be sent to one or more external Syslog server(s). By default, audit records are stored in the Loki Grafana Microservice, and continue to do so even if Syslog connections are configured. Each server audit record will be sent to each configured Syslog connection.

Refer to Log Forwarding for instructions on configuring a new connection to a Syslog server on the CipherTrust Manager console. Refer to the API guide, CLI Guide, or the GUI for more details.

A Syslog connection can either use UDP or TCP + TLS as the transport protocol. When TCP + TLS is used a trusted CA certificate in PEM format must also be provided.

All Syslog messages are generated with severity notice and facility local0.

An example entry looks as follows:

2019-08-12 06:22:16 keysecure1 KeySecure: <134>1 2019-08-12 06:22:16 keysecure KeySecure - b7999852-dd64-46e9-b924-c3999eca9fad [msg="Update Syslog Connection" sev="6" details="'Update Syslog Connection' succeeded”
({"createdAt":"2019-08-12T06:22:16.113081Z", "details":{"id":"e847e529-d331-45f9-a494-83ee7ce6ab69"},"message":"Update Syslog Connection","service":"kylo","success":true,"username":"admin","severity":"info","lineage":"","source":""})"]

The first time stamp is generated and added by Syslog and the second time stamp is the time of the actual audit record. The time stamps use the UTC time zone.

In a multi-node clustered environment, the Syslog connection configurations are automatically synchronized and each node is aware of all Syslog servers. The Syslog message is sent from the currently active node. This means that if an event that results in a server audit record is performed on node 1, the Syslog message originates from node 1. Similarly, if a server audit event is performed on node 2, the Syslog message originates from node 2.

Local Database Audit Logging

Local database logging of server and client audit records is disabled by default, to reduce cluster traffic and disk usage. Local database logging is deprecated; for most deployments, we recommend to view records on Loki Grafana microservice, and to enable logging using a remote Syslog server.

If local database audit logging is enabled, you can list and query records on the local database.

Enabling and disabling Local Database Audit Logging

If you have a production system and cluster with a limited load, you might like to enable local database logging, which is disabled by default.

Enabling Example

ksctl properties modify -n ENABLE_RECORDS_DB_STORE -p true

The Server Audit Records database table is automatically trimmed when it reaches 10 million records in size. If a permanent record is required, syslog should be used and the audit records stored elsewhere.

If you wish to later disable local database logging, run:

ksctl properties modify -n ENABLE_RECORDS_DB_STORE -p false

Listing and Querying Records With Local Database Logging

If your server and client audit records are logged to the local database, you can list server audit records with ksctl records list and client audit records with ksctl client-records list.

The response looks like:

{
  "skip": 0,
  "limit": 10,
  "total": 40,
  "Resources": [
    {
    "id": "18ef98f1-0704-4dec-a6f0-e87a581ce52f",
    "uri": "kylo:kylo:audit:records:18ef98f1-0704-4dec-a6f0-e87a581ce52f",
    "account": "kylo:kylo:admin:accounts:kylo",
    "application": "system:system:admin:applications:system",
    "devAccount": "kylo:kylo:admin:accounts:kylo",
    "createdAt": "2018-05-22T18:34:05.239823Z",
    "message": "Create Token",
    "service": "kylo",
    "requestId": "45a8dcef-0b9e-44e2-b3e1-51b3d2ef99ca",
    "instanceId": "",
    "instanceTime": "0001-01-01T00:00:00Z",
    "instanceTz": "",
    "success": true,
    "username": ""
    }
  ]
}

There will be a number of records returned (10 shown in the example above). Most have been removed in the above example for space reasons. Note that the response indicates that there are 40 records available ("total": 40,). These records can be returned with options in the ksctl records and ksctl client-records commands to filter which records are returned. Type the following commands to get more information:

ksctl records list --help
ksctl client-records list --help

Configuring alarm triggers based on a record

CipherTrust Manager can be configured to trigger one or more alarms based on the content of a record.

Alarm configurations apply to both records stored in the Loki Grafana microservice, and records stored in the database, if enabled.

To create an alarm configuration:

1. Log on to the CipherTrust Manager console as administrator.

2. In the left pane, click Records. Click Loki Audit Records.

3. Click the Alarm Configurations tab.

4. Click Configure Alarm Configuration. The Alarm Config - Create dialog box is displayed.
   Specify the following details:

   1. Severity: Defines the nature of the alarm. It can be Info, Error, Critical, or Warning.

      Note

      This value is independent from the severity included with server records. If you want an alarm to inherit the severity of a server record, you can create the alarm configuration using the REST API or CLI and omit the severity option.

   2. Name: Name of the alarm.

   3. ErrorThreshold: The maximum number of times the trigger condition is met in the specified time Interval before the alarm is raised.

   4. Interval: The time duration in seconds, in which if the trigger condition is met for a defined number of times (Error Threshold value) , the alarm is raised.

   5. Description: A description for the alarm.

   6. Condition: Conditional properties/events that triggers an alarm.

      Note

      Multiple alarms may be triggered using a single record.

Example (For Windows)

ksctl records alarm-configs create --name "Weak RSA Key" --description "RSA key should be 2048 bits or higher" --severity "critical" --condition "input.success; input.message == \"Create Key\"; input.details.algorithm == \"RSA\"; input.details.size <= 1024"

Example (For Linux)

ksctl records alarm-configs create --name "Weak RSA Key" --description "RSA key should be 2048 bits or higher" --severity critical --condition 'input.success; input.message == "Create Key"; input.details.algorithm == "RSA"; input.details.size <= 1024'

Response (For Windows and Linux):

{
  "id": "4bf23269-3985-42f3-995c-8f3c8f8b6543",
  "uri": "kylo:kylo:audit:alarm-configs:4bf23269-3985-42f3-995c-8f3c8f8b6543",
  "account": "kylo:kylo:admin:accounts:kylo",
  "application": "ncryptify:gemalto:admin:apps:kylo",
  "devAccount": "ncryptify:gemalto:admin:accounts:gemalto",
  "createdAt": "2019-08-27T14:37:19.145946719Z",
  "name": "Weak RSA Key",
  "condition": "input.success; input.message == \"Create Key\"; input.details.algorithm == \"RSA\"; input.details.size <= 1024",
  "description": "RSA key should be 2048 bits or higher",
  "severity": "critical"
}

To trigger this alarm, create a weak RSA key:

ksctl keys create --name mysmallkey --alg RSA --size 1024

The alarm should be triggered and this can be confirmed by listing the alarms:

ksctl alarms list --state on

Here is an abbreviated response for brevity:

{
  "resources": [
    {
      "name": "Weak RSA Key",
      "state": "on",
      "description": "RSA key should be 2048 bits or higher",
      "severity": "critical",
      "details": {
        "details": {
          "id": "f2ec44f6977edf2d6ff7fa327b5499301732dc6b395284a71e47ac0346cf72d6",
          "uri": "kylo:kylo:vault:keys:mysmallkey-v0",
          "name": "mysmallkey",
          "size": 1024,
          "ownerId": "local|fa0e4dfa-e6fd-413b-a4a0-48c356cf0557",
          "algorithm": "RSA",
          "usageMask": 3,
          "objectType": "Private Key"
        },
        "message": "Create Key",
        "success": true,
      }
    }
  ]
}

Creating Alarm Trigger for Certificate Expiration

To create an alarm trigger (and receive email notifications) for certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Certificate Expiry Check".

After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a certificate expiration related record is logged. The emails will contain the certificate IDs, expiration dates, and the domain name.

Creating Alarm Trigger for Client Certificate Expiration

To create an alarm trigger (and receive email notifications) for client certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Client Certificate Expiry Check".

After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a client certificate expiration related record is logged.
The emails will contain the following information:

• Client name

• Client type if available

• Whether certificate is expired

• Expiration duration in days (if expired flag is false) or expired duration in days (if expired flag is true)

• Domain name

Creating Alarm Trigger for Connection Certificate Expiration

To create an alarm trigger (and receive email notifications) for the connection certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Connection Certificate Expiry Check".

After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a connection certificate expiration related record is logged.
The emails will contain the following information:

• Domain name

• Whether certificate is expired (expired flag set to true/false)

• Expiration duration in days (days_since_expired, if expired flag is true) and (days_to_expire, if expired flag is false)

Creating Alarm Trigger for Interface Certificate Expiration

To create an alarm trigger (and receive email notifications) for the interface certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Interface Certificate Expiry Check".

After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a Interface certificate expiration related record is logged.
The emails will contain the following information:

• Interface name

• Whether certificate is expired (expired flag set to true/false)

• Expiration duration in days (days_since_expired, if expired flag is true) and (days_to_expire, if expired flag is false)

Creating Alarm Trigger for System Restart

To create an alarm trigger for a system restart, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Restart Services" input.service == "sheepdog"

This alarm will trigger to inform the users about the reboot operation on the CipherTrust Manager.

Creating Alarm Trigger for Service Restart

To create an alarm trigger for a service restart, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Restart Services"

This alarm will trigger to inform the users about a restart performed on a service.

Creating Alarm Trigger to Enable Prometheus Metrics

To create an alarm trigger to enable Prometheus metrics on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Enable Prometheus Metrics Collection" input.service == "kylo"

This alarm will trigger to inform the users that Prometheus metrics are enabled on the CipherTrust Manager.

Creating Alarm Trigger to Disable Prometheus Metrics

To create an alarm trigger to disable Prometheus metrics on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Disable Prometheus Metrics Collection" input.service == "kylo"

This alarm will trigger to inform the users that Prometheus metrics are disabled on the CipherTrust Manager.

Creating Alarm Trigger for Certificate Expiration Warning

To create an alarm trigger for a certificate expiration warning, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Following certificates are expired or going to expire in a few days. To avoid a possible interruption of your service, please renew your certificates." input.message == "Certificate Expiry Check" input.service == "naboo"

This alarm will trigger to inform the user about the certificates that are going to be expired in near future or have already expired.

Creating Alarm Trigger for Multiple Login Failure Occurrences

To create an alarm trigger for multiple login failures occurrences on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

"input.message == "error getting client/user: Wrong username or password." input.message == "Create Token"

Creating Alarm Trigger for Failed System Backup using SCP

To create an alarm trigger for failed system backup using SCP, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.details.status == "failed" input.message == "SCP Backup"

This alarm will trigger to inform the user about the failure of securely copying a backup to the set destination.

Creating Alarm Trigger for Unauthorized Key Usage

To create an alarm trigger for unauthorized key usage on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "read on target is not authorized: verdict included conditions, which are not supported: ReadKey" input.service == "minerva"

This alarm will trigger to inform the user about an unauthorized attempt to access an existing key on the CipherTrust Manager.

Creating Alarm Trigger when Disk/Key Storage is Full

To create an alarm trigger when the disk/key storage on the CipherTrust Manager is full, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Disk Full" input.details.previous_state == "off" input.details.service == "Sheepdog" input.details.state == "on" input.service == "Sheepdog"

This alarm will trigger to inform the user that storage on the CipherTrust Manager has been consumed more than the configured threshold limit of 80%.

Creating Alarm Trigger for Unreachable HSM

To create an alarm trigger when HSM is offline, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "HSM is offline" input.details.name == "HSM Offline" input.details.service == "Darkstar" input.details.source == "ciphertrust" input.details.source_type == "server_record" input.service == "Darkstar" input.source == "ciphertrust"

This alarm will trigger to inform the user that the CipherTrust Manager is unable to access the HSM.

Creating Alarm Trigger when Cluster Node is Unavailable

To create an alarm trigger when a node in a cluster goes down, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:

input.message == "Cluster Node Down" input.details.service == "dbmgr" input.details.source_type == "server_record" input.service == "dbmgr"

This alarm will trigger to inform the user about the unavailability of a node within the cluster.