Records

Audit records are logs of auditable events viewable in CipherTrust Manager in a user-friendly JSON format.

Server records contain operations performed by the CipherTrust Manager. You can also view the audit records that are uploaded from the CipherTrust Manager Clients (for example, CTE). Consult documentation for a specific connector for information on configuring and interpreting client records. The Logs chapter of CTE Agent for Linux Advanced Configuration and Integration Guide contains details on CTE client record format and effects codes.

Audit records can be locally fetched and searched using the CipherTrust Manager REST API, CLI, or the GUI. The records available through these queries are scoped to the domain permissions of the logged in user.

Both server and client records can be configured to raise alarms.

There are three options to store and display records:

• Loki Grafana Microservice

• Log Forwarding

• Local Database Logging

Loki Grafana Microservice Option

Records for the local node are automatically logged to an internal Loki Grafana log aggregation microservice, where they can be viewed and queried. This option has the best performance locally. In the GUI, Loki records are visible under Records > Loki Audit Records. The retention period for records is configurable, with a default value of 240 hours. These records are not replicated in cluster.

Log Forwarding Options

Records can also be sent to an external logging system through a log forwarder. Supported log forwarder types are Elasticsearch, Loki, and Syslog server. These options allow you to view records for a cluster in aggregate.

Local Database Logging Option

To view database audit records for CipherTrust Manager activity, you must first enable local database logging. Then, in the GUI, go to Records > Server Records. To view client records in the GUI, go to Records > Client Records. The Server Audit Records database table is automatically trimmed when it reaches 10 million records in size.

Server Records Parameters

Loki Grafana Microservice

You can list and query records on the Loki Grafana Microservice. As well, you can configure the retention period, after which records are deleted.

Listing and Querying Records With Loki Grafana Microservice

You can list records in the Loki Grafana microservice with ksctl loki query_range --server-audit-records and ksctl loki query_range --client-audit-records. The available records depend on the configured retention period after which records are deleted. The default retention period is 240 hours.

You can also query both server audit records and client audit records from the local CipherTrust Manager node to return records during a specified time range.

The query parameters follow Loki Grafana's LogQL syntax for doing a query over a specified time range. Use ksctl loki query_range commands to input parameters for limit, start, end, step, interval, and direction.

Saved Queries With Loki Grafana Microservice

The CipherTrust Manager web console UI allows you to save and load queries for convenience.

We also provide two public server records queries available to load for Google CSE Records and Google Cloud EKM Records, which are CipherTrust Cloud Key Manager (CCKM) integrations.

To save a query:

1. Login as an Application administrator, such as the admin user.

2. Navigate to Records>Loki Audit Records. Select Server Records or Client Records.

3. Select Advanced Query.

   A text box appears.

4. Input a query following Loki Grafana's LogQL syntax.

5. Optionally, select Apply Query to view which records are returned.

6. Select Save Query.

   The Save Query popup window appears.

7. Provide a Name and review the Query Preview.

   Note

   You can only save the query as Private.

8. Select Save to finalize.

To load a query:

1. Login as an Application administrator, such as the admin user.

2. Navigate to Records>Loki Audit Records. Select Server Records or Client Records.

3. Select Advanced Query.

   A text box appears for query input.

4. Select Load Saved Query.

   The Load Saved Query popup appears.

5. Select one of the saved queries on the left.

   Two public queries for CCKM integrations, Google CSE Records and Google Cloud EKM Records, are available, as well as any previously saved private queries.

6. Select Load.

   The query is loaded and applied.

Configuring Retention Period for Loki Grafana Microservice

By default, all records generated since the CipherTrust Manager is deployed are retained in Loki. To preserve disk space, users in the 'admin' group can set a retention period for which Loki logs are retained.

The retention period has the following characteristics:

• The minimum value is 24 hours.

• A valid value must be a multiple of 24 hours.

• The retention period must be a string that can be parsed using Go library’s time.Duration type, such as 24h.

To set the Loki configuration retention period:

ksctl loki config modify --retention-time "<desired_retention_time>"

To view the Loki configuration retention period:

ksctl loki config get

Configuring remote Syslog server

Records can optionally be sent to one or more external Syslog server(s). By default, audit records are stored in the Loki Grafana Microservice, and continue to do so even if Syslog connections are configured. Each server audit record will be sent to each configured Syslog connection.

Refer to Log Forwarding for instructions on configuring a new connection to a Syslog server on the CipherTrust Manager console. Refer to the API guide, CLI Guide, or the GUI for more details.

A Syslog connection can either use UDP or TCP + TLS as the transport protocol. When TCP + TLS is used a trusted CA certificate in PEM format must also be provided.

All Syslog messages are generated with severity notice and facility local0.

An example entry looks as follows:

2019-08-12 06:22:16 keysecure1 KeySecure: <134>1 2019-08-12 06:22:16 keysecure KeySecure - b7999852-dd64-46e9-b924-c3999eca9fad [msg="Update Syslog Connection" sev="6" details="'Update Syslog Connection' succeeded”
({"createdAt":"2019-08-12T06:22:16.113081Z", "details":{"id":"e847e529-d331-45f9-a494-83ee7ce6ab69"},"message":"Update Syslog Connection","service":"kylo","success":true,"username":"admin","severity":"info","lineage":"","source":""})"]

The first time stamp is generated and added by Syslog and the second time stamp is the time of the actual audit record. The time stamps use the UTC time zone.

In a multi-node clustered environment, the Syslog connection configurations are automatically synchronized and each node is aware of all Syslog servers. The Syslog message is sent from the currently active node. This means that if an event that results in a server audit record is performed on node 1, the Syslog message originates from node 1. Similarly, if a server audit event is performed on node 2, the Syslog message originates from node 2.

Local Database Audit Logging

Local database logging of server and client audit records is disabled by default, to reduce cluster traffic and disk usage. Local database logging is deprecated; for most deployments, we recommend to view records on Loki Grafana microservice, and to enable logging using a remote Syslog server.

If local database audit logging is enabled, you can list and query records on the local database.

Enabling and disabling Local Database Audit Logging

If you have a production system and cluster with a limited load, you might like to enable local database logging, which is disabled by default.

Enabling Example

ksctl properties modify -n ENABLE_RECORDS_DB_STORE -p true

The Server Audit Records database table is automatically trimmed when it reaches 10 million records in size. If a permanent record is required, syslog should be used and the audit records stored elsewhere.

If you wish to later disable local database logging, run:

ksctl properties modify -n ENABLE_RECORDS_DB_STORE -p false

Listing and Querying Records With Local Database Logging

If your server and client audit records are logged to the local database, you can list server audit records with ksctl records list and client audit records with ksctl client-records list.

The response looks like:

{
  "skip": 0,
  "limit": 10,
  "total": 40,
  "Resources": [
    {
    "id": "18ef98f1-0704-4dec-a6f0-e87a581ce52f",
    "uri": "kylo:kylo:audit:records:18ef98f1-0704-4dec-a6f0-e87a581ce52f",
    "account": "kylo:kylo:admin:accounts:kylo",
    "application": "system:system:admin:applications:system",
    "devAccount": "kylo:kylo:admin:accounts:kylo",
    "createdAt": "2018-05-22T18:34:05.239823Z",
    "message": "Create Token",
    "service": "kylo",
    "requestId": "45a8dcef-0b9e-44e2-b3e1-51b3d2ef99ca",
    "instanceId": "",
    "instanceTime": "0001-01-01T00:00:00Z",
    "instanceTz": "",
    "success": true,
    "username": ""
    }
  ]
}

There will be a number of records returned (10 shown in the example above). Most have been removed in the above example for space reasons. Note that the response indicates that there are 40 records available ("total": 40,). These records can be returned with options in the ksctl records and ksctl client-records commands to filter which records are returned. Type the following commands to get more information:

ksctl records list --help
ksctl client-records list --help

Configuring alarm triggers based on a record

CipherTrust Manager can be configured to trigger one or more alarms based on the content of a record.

Alarm configurations apply to both records stored in the Loki Grafana microservice, and records stored in the database, if enabled.

To create an alarm configuration:

1. Log on to the CipherTrust Manager console as administrator.

2. In the left pane, click Records. Click Loki Audit Records.

3. Click the Alarm Configurations tab.

4. Click Configure Alarm Configuration. The Alarm Config - Create dialog box is displayed.
   Specify the following details:

   1. Severity: Defines the nature of the alarm. It can be Info, Error, Critical, or Warning.

      Note

      This value is independent from the severity included with server records. If you want an alarm to inherit the severity of a server record, you can create the alarm configuration using the REST API or CLI and omit the severity option.

   2. Name: Name of the alarm.

   3. ErrorThreshold: The maximum number of times the trigger condition is met in the specified time Interval before the alarm is raised.

   4. Interval: The time duration in seconds, in which if the trigger condition is met for a defined number of times (Error Threshold value) , the alarm is raised.

   5. Description: A description for the alarm.

   6. Condition: Conditional properties/events that triggers an alarm.

      Note

      Multiple alarms may be triggered using a single record.

Example (For Windows)

ksctl records alarm-configs create --name "Weak RSA Key" --description "RSA key should be 2048 bits or higher" --severity "critical" --condition "input.success; input.message == \"Create Key\"; input.details.algorithm == \"RSA\"; input.details.size <= 1024"

Example (For Linux)

ksctl records alarm-configs create --name "Weak RSA Key" --description "RSA key should be 2048 bits or higher" --severity critical --condition 'input.success; input.message == "Create Key"; input.details.algorithm == "RSA"; input.details.size <= 1024'

Response (For Windows and Linux):

{
  "id": "4bf23269-3985-42f3-995c-8f3c8f8b6543",
  "uri": "kylo:kylo:audit:alarm-configs:4bf23269-3985-42f3-995c-8f3c8f8b6543",
  "account": "kylo:kylo:admin:accounts:kylo",
  "application": "ncryptify:gemalto:admin:apps:kylo",
  "devAccount": "ncryptify:gemalto:admin:accounts:gemalto",
  "createdAt": "2019-08-27T14:37:19.145946719Z",
  "name": "Weak RSA Key",
  "condition": "input.success; input.message == \"Create Key\"; input.details.algorithm == \"RSA\"; input.details.size <= 1024",
  "description": "RSA key should be 2048 bits or higher",
  "severity": "critical"
}

To trigger this alarm, create a weak RSA key:

ksctl keys create --name mysmallkey --alg RSA --size 1024

The alarm should be triggered and this can be confirmed by listing the alarms:

ksctl alarms list --state on

Here is an abbreviated response for brevity:

{
  "resources": [
    {
      "name": "Weak RSA Key",
      "state": "on",
      "description": "RSA key should be 2048 bits or higher",
      "severity": "critical",
      "details": {
        "details": {
          "id": "f2ec44f6977edf2d6ff7fa327b5499301732dc6b395284a71e47ac0346cf72d6",
          "uri": "kylo:kylo:vault:keys:mysmallkey-v0",
          "name": "mysmallkey",
          "size": 1024,
          "ownerId": "local|fa0e4dfa-e6fd-413b-a4a0-48c356cf0557",
          "algorithm": "RSA",
          "usageMask": 3,
          "objectType": "Private Key"
        },
        "message": "Create Key",
        "success": true,
      }
    }
  ]
}

Creating Alarm Trigger for Certificate Expiration

If it is desired to create an Alarm Trigger (and receive Email Notifications) for these events, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Certificate Expiry Check".

After saving this Alarm Configuration, the recipients in the email address list will receive email notifications every time a certificate expiration related record is logged. The emails will contain the certificate IDs, expiration dates, and the domain name.

Creating Alarm Trigger for Client Certificate Expiration

If it is desired to create an Alarm Trigger (and receive Email Notifications) for these events, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Client Certificate Expiry Check".

After saving this Alarm Configuration, the recipients in the email address list will receive email notifications every time a client certificate expiration related record is logged.
The emails will contain the following information:

• Client name

• Client type if available

• Whether certificate is expired

• Expiration duration in days (if expired flag is false) or expired duration in days (if expired flag is true)

• Domain name

Creating Alarm Trigger for Connection Certificate Expiration

If it is desired to create an Alarm Trigger (and receive Email Notifications) for these events, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Connection Certificate Expiry Check".

After saving this Alarm Configuration, the recipients in the email address list will receive email notifications every time a client certificate expiration related record is logged.
The emails will contain the following information:

• Domain name

• Whether certificate is expired (expired flag set to true/false)

• Expiration duration in days (days_since_expired, if expired flag is true) and (days_to_expire, if expired flag is false)

Creating Alarm Trigger for Interface Certificate Expiration

If it is desired to create an Alarm Trigger (and receive Email Notifications) for these events, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Interface Certificate Expiry Check".

After saving this Alarm Configuration, the recipients in the email address list will receive email notifications every time a client certificate expiration related record is logged.
The emails will contain the following information:

• Interface name

• Whether certificate is expired (expired flag set to true/false)

• Expiration duration in days (days_since_expired, if expired flag is true) and (days_to_expire, if expired flag is false)