Records
Audit records are logs of auditable events viewable in CipherTrust Manager in a user-friendly JSON format.
Server records contain operations performed by the CipherTrust Manager. You can also view the audit records that are uploaded from the CipherTrust Manager Clients (for example, CTE). Consult documentation for a specific connector for information on configuring and interpreting client records. The Logs chapter of CTE Agent for Linux Advanced Configuration and Integration Guide contains details on CTE client record format and effects codes.
Audit records can be locally fetched and searched using the CipherTrust Manager REST API, CLI, or the GUI. The records available through these queries are scoped to the domain permissions of the logged in user.
Both server and client records can be configured to raise alarms.
There are three options to store and display records:
Loki Grafana Microservice
Log Forwarding
Local Database Logging
Loki Grafana Microservice Option
Records for the local node are automatically logged to an internal Loki Grafana log aggregation microservice, where they can be viewed and queried. This option has the best performance locally. In the GUI, Loki records are visible under Records > Loki Audit Records. The retention period for records is configurable, with a default value of 240 hours. These records are not replicated in cluster.
Log Forwarding Options
Records can also be sent to an external logging system through a log forwarder. Supported log forwarder types are Elasticsearch, Loki, and Syslog server. These options allow you to view records for a cluster in aggregate.
Local Database Logging Option
Note
To reduce cluster traffic and disk usage, local database audit logging is disabled by default. Local database logging is deprecated, and we recommend using Loki Grafana microservice and/or enabling forwarding to a remote Syslog.
To view database audit records for CipherTrust Manager activity, you must first enable local database logging. Then, in the GUI, go to Records > Server Records. To view client records in the GUI, go to Records > Client Records. The Server Audit Records database table is automatically trimmed when it reaches 10 million records in size.
Server Records Parameters
Parameters (Columns) | Description |
---|---|
Status | Indicates whether an operation was successful or failed. |
Event Created | Date and time when the event was created. Time is in UTC time zone. |
Severity | Severity of the record. Severity types are: • INFO • CRITICAL • WARNING • ERROR |
Message | A short description of the operation performed. |
Service | Indicates which internal microservice performed the operation. This is mostly valuable for troubleshooting purposes in conjunction with customer support. |
Details | Details associated with the operation performed. These include user_id and client_id to show which entity performed the operation. |
Loki Grafana Microservice
You can list and query records on the Loki Grafana Microservice, and download the records to a file. As well, you can configure the retention period, after which records are deleted.
Note
All the CCKM-managed AWS/Azure keys that are going to expire in the next 10 days are displayed under the Loki Audit Records. After the CipherTrust Manager starts, a back-end job runs every 24 hours to:
Fetch the list of AWS/Azure keys going to expire in the next 10 days.
Log records for a batch of every 20 keys.
These records are available under Records > Loki Audit Records > Server Records.
Listing and Querying Records With Loki Grafana Microservice
You can query both server audit records and client audit records from the local CipherTrust Manager node to return records. The available records depend on the configured retention period after which records are deleted. The default retention period is 240 hours.
Note
Only data for the local node is logged to the Loki Grafana microservice, not any other cluster nodes.
View and Query Records in the GUI
Application administrators, such as the admin
user can view and query records. There is a basic query and an advanced query to limit which Loki audit records are displayed.
To view records, navigate to Records>Loki Audit Records. Select Server Records or Client Records.
Basic Query
Select the filter icon () to filter on particular fields or within a specified time range. The following parameters are available to filter on:
Severity
Status
By
Source
Client IP
Message
Max Records
Date and Time
Start Date
End Date
Select Apply Query to view which records are returned.
Advanced Query
Select Advanced Query to input a query following Loki Grafana's LogQL syntax. If you have applied a basic query, it is already inputted in LogQL syntax. There are options to save and load queries from the Advanced Query menu.
Select Apply Query to view which records are returned.
List and Query Records with ksctl
Application administrators, such as the admin
user can list records in the Loki Grafana microservice with ksctl loki query_range --server-audit-records
and ksctl loki query_range --client-audit-records
.
The query parameters follow Loki Grafana's LogQL syntax for doing a query over a specified time range. Use ksctl loki query_range
commands to input parameters for limit
, start
, end
, step
, interval
, and direction
.
Save Queries With Loki Grafana Microservice
The CipherTrust Manager web console UI allows you to save and load queries for convenience.
We also provide two public server records queries available to load for Google CSE Records and Google Cloud EKM Records, which are CipherTrust Cloud Key Manager (CCKM) integrations.
To save a query:
Login as an Application administrator, such as the
admin
user.Navigate to Records>Loki Audit Records. Select Server Records or Client Records.
Select Advanced Query.
A text box appears.
Input a query following Loki Grafana's LogQL syntax.
Optionally, select Apply Query to view which records are returned.
Select Save Query.
The Save Query popup window appears.
Provide a Name and review the Query Preview.
Note
You can only save the query as Private.
Select Save to finalize.
To load a query:
Login as an Application administrator, such as the
admin
user.Navigate to Records>Loki Audit Records. Select Server Records or Client Records.
Select Advanced Query.
A text box appears for query input.
Select Load Saved Query.
The Load Saved Query popup appears.
Select one of the saved queries on the left.
Two public queries for CCKM integrations, Google CSE Records and Google Cloud EKM Records, are available, as well as any previously saved private queries.
Select Load.
The query is loaded and applied.
Download Loki Audit Records to a File
You can download the Loki Audit Records displayed in the CipherTrust Manager web console UI to a file.
To download the records to a file:
Login as an Application administrator, such as the
admin
user.Navigate to Records>Loki Audit Records. Select Server Records or Client Records.
Adjust the query as desired, to limit which Loki audit records will be downloaded.
Click Download Logs in the upper right corner.
The logs are downloaded to a file
loki_audit_log.txt
.
Configuring Retention Period for Loki Grafana Microservice
By default, all records generated since the CipherTrust Manager is deployed are retained in Loki. To preserve disk space, users in the 'admin' group can set a retention period for which Loki logs are retained.
The retention period has the following characteristics:
The minimum value is 24 hours.
A valid value must be a multiple of 24 hours.
The retention period must be a string that can be parsed using Go library’s
time.Duration
type, such as 24h.
To set the Loki configuration retention period:
ksctl loki config modify --retention-time "<desired_retention_time>"
To view the Loki configuration retention period:
ksctl loki config get
Configuring remote Syslog server
Records can optionally be sent to one or more external Syslog server(s). By default, audit records are stored in the Loki Grafana Microservice, and continue to do so even if Syslog connections are configured. Each server audit record will be sent to each configured Syslog connection.
Refer to Log Forwarding for instructions on configuring a new connection to a Syslog server on the CipherTrust Manager console. Refer to the API guide, CLI Guide, or the GUI for more details.
Note
Upgraded CipherTrust Manager instances can have existing syslog connections through Admin Settings, which continue to be supported. Syslog servers configured as log forwarders can forward client audit records, while syslog servers configured through Admin Settings cannot.
A Syslog connection can either use UDP or TCP + TLS as the transport protocol. When TCP + TLS is used a trusted CA certificate in PEM format must also be provided.
All Syslog messages are generated with severity notice and facility local0.
An example entry looks as follows:
2019-08-12 06:22:16 keysecure1 KeySecure: <134>1 2019-08-12 06:22:16 keysecure KeySecure - b7999852-dd64-46e9-b924-c3999eca9fad [msg="Update Syslog Connection" sev="6" details="'Update Syslog Connection' succeeded”
({"createdAt":"2019-08-12T06:22:16.113081Z", "details":{"id":"e847e529-d331-45f9-a494-83ee7ce6ab69"},"message":"Update Syslog Connection","service":"kylo","success":true,"username":"admin","severity":"info","lineage":"","source":""})"]
The first time stamp is generated and added by Syslog and the second time stamp is the time of the actual audit record. The time stamps use the UTC time zone.
In a multi-node clustered environment, the Syslog connection configurations are automatically synchronized and each node is aware of all Syslog servers. The Syslog message is sent from the currently active node. This means that if an event that results in a server audit record is performed on node 1, the Syslog message originates from node 1. Similarly, if a server audit event is performed on node 2, the Syslog message originates from node 2.
Note
The Syslog connection configuration can take up to five minutes to apply to all nodes in the cluster.
Local Database Audit Logging
Local database logging of server and client audit records is disabled by default, to reduce cluster traffic and disk usage. Local database logging is deprecated; for most deployments, we recommend to view records on Loki Grafana microservice, and to enable logging using a remote Syslog server.
If local database audit logging is enabled, you can list and query records on the local database.
Enabling and disabling Local Database Audit Logging
If you have a production system and cluster with a limited load, you might like to enable local database logging, which is disabled by default.
Enabling Example
ksctl properties modify -n ENABLE_RECORDS_DB_STORE -p true
Note
This is a cluster wide setting and only needs to be run against one node in the cluster.
The Server Audit Records database table is automatically trimmed when it reaches 10 million records in size. If a permanent record is required, syslog should be used and the audit records stored elsewhere.
If you wish to later disable local database logging, run:
ksctl properties modify -n ENABLE_RECORDS_DB_STORE -p false
Listing and Querying Records With Local Database Logging
If your server and client audit records are logged to the local database, you can list server audit records with ksctl records list
and client audit records with ksctl client-records list
.
The response looks like:
{
"skip": 0,
"limit": 10,
"total": 40,
"Resources": [
{
"id": "18ef98f1-0704-4dec-a6f0-e87a581ce52f",
"uri": "kylo:kylo:audit:records:18ef98f1-0704-4dec-a6f0-e87a581ce52f",
"account": "kylo:kylo:admin:accounts:kylo",
"application": "system:system:admin:applications:system",
"devAccount": "kylo:kylo:admin:accounts:kylo",
"createdAt": "2018-05-22T18:34:05.239823Z",
"message": "Create Token",
"service": "kylo",
"requestId": "45a8dcef-0b9e-44e2-b3e1-51b3d2ef99ca",
"instanceId": "",
"instanceTime": "0001-01-01T00:00:00Z",
"instanceTz": "",
"success": true,
"username": ""
}
]
}
There will be a number of records returned (10 shown in the example above). Most have been removed in the above example for space reasons. Note that the response indicates that there are 40 records available ("total": 40,). These records can be returned with options in the ksctl records and ksctl client-records commands to filter which records are returned. Type the following commands to get more information:
ksctl records list --help
ksctl client-records list --help
Configuring alarm triggers based on a record
CipherTrust Manager can be configured to trigger one or more alarms based on the content of a record.
Alarm configurations apply to both records stored in the Loki Grafana microservice, and records stored in the database, if enabled.
To create an alarm configuration:
Log on to the CipherTrust Manager console as administrator.
In the left pane, click Records. Click Loki Audit Records.
Click the Alarm Configurations tab.
Click Configure Alarm Configuration. The Alarm Config - Create dialog box is displayed.
Specify the following details:Severity: Defines the nature of the alarm. It can be Info, Error, Critical, or Warning.
Note
This value is independent from the severity included with server records. If you want an alarm to inherit the severity of a server record, you can create the alarm configuration using the REST API or CLI and omit the severity option.
Name: Name of the alarm.
ErrorThreshold: The maximum number of times the trigger condition is met in the specified time Interval before the alarm is raised.
Interval: The time duration in seconds, in which if the trigger condition is met for a defined number of times (Error Threshold value) , the alarm is raised.
Description: A description for the alarm.
Condition: Conditional properties/events that triggers an alarm.
Note
Multiple alarms may be triggered using a single record.
Example (For Windows)
Note
In Windows environment, ensure to use backslash ('\') as an escaped character with double quotes (\").
ksctl records alarm-configs create --name "Weak RSA Key" --description "RSA key should be 2048 bits or higher" --severity "critical" --condition "input.success; input.message == \"Create Key\"; input.details.algorithm == \"RSA\"; input.details.size <= 1024"
Example (For Linux)
ksctl records alarm-configs create --name "Weak RSA Key" --description "RSA key should be 2048 bits or higher" --severity critical --condition 'input.success; input.message == "Create Key"; input.details.algorithm == "RSA"; input.details.size <= 1024'
Response (For Windows and Linux):
{
"id": "4bf23269-3985-42f3-995c-8f3c8f8b6543",
"uri": "kylo:kylo:audit:alarm-configs:4bf23269-3985-42f3-995c-8f3c8f8b6543",
"account": "kylo:kylo:admin:accounts:kylo",
"application": "ncryptify:gemalto:admin:apps:kylo",
"devAccount": "ncryptify:gemalto:admin:accounts:gemalto",
"createdAt": "2019-08-27T14:37:19.145946719Z",
"name": "Weak RSA Key",
"condition": "input.success; input.message == \"Create Key\"; input.details.algorithm == \"RSA\"; input.details.size <= 1024",
"description": "RSA key should be 2048 bits or higher",
"severity": "critical"
}
To trigger this alarm, create a weak RSA key:
ksctl keys create --name mysmallkey --alg RSA --size 1024
The alarm should be triggered and this can be confirmed by listing the alarms:
ksctl alarms list --state on
Here is an abbreviated response for brevity:
{
"resources": [
{
"name": "Weak RSA Key",
"state": "on",
"description": "RSA key should be 2048 bits or higher",
"severity": "critical",
"details": {
"details": {
"id": "f2ec44f6977edf2d6ff7fa327b5499301732dc6b395284a71e47ac0346cf72d6",
"uri": "kylo:kylo:vault:keys:mysmallkey-v0",
"name": "mysmallkey",
"size": 1024,
"ownerId": "local|fa0e4dfa-e6fd-413b-a4a0-48c356cf0557",
"algorithm": "RSA",
"usageMask": 3,
"objectType": "Private Key"
},
"message": "Create Key",
"success": true,
}
}
]
}
Creating Alarm Trigger for Certificate Expiration
To create an alarm trigger (and receive email notifications) for certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Certificate Expiry Check"
.
Note
The action Certificate Expiry Check can be created in the root domain only.
After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a certificate expiration related record is logged. The emails will contain the certificate IDs, expiration dates, and the domain name.
Creating Alarm Trigger for Client Certificate Expiration
To create an alarm trigger (and receive email notifications) for client certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Client Certificate Expiry Check"
.
Note
The action Client Certificate Expiry Check can be created in the root domain only.
After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a client certificate expiration related record is logged.
The emails will contain the following information:
Client name
Client type if available
Whether certificate is expired
Expiration duration in days (if
expired
flag isfalse
) or expired duration in days (ifexpired
flag istrue
)Domain name
Creating Alarm Trigger for Connection Certificate Expiration
To create an alarm trigger (and receive email notifications) for the connection certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Connection Certificate Expiry Check"
.
Note
The action Connection Certificate Expiry Check can be created in the root domain only.
After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a connection certificate expiration related record is logged.
The emails will contain the following information:
Domain name
Whether certificate is expired (
expired
flag set totrue/false
)Expiration duration in days (
days_since_expired
, ifexpired
flag istrue
) and (days_to_expire
, ifexpired
flag isfalse
)
Creating Alarm Trigger for Interface Certificate Expiration
To create an alarm trigger (and receive email notifications) for the interface certificate expiration, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as input.message == "Interface Certificate Expiry Check"
.
Note
The action Interface Certificate Expiry Check can be created in the root domain only.
After saving this alarm configuration, the recipients in the email address list will receive email notifications every time a Interface certificate expiration related record is logged.
The emails will contain the following information:
Interface name
Whether certificate is expired (
expired
flag set totrue/false
)Expiration duration in days (
days_since_expired
, ifexpired
flag istrue
) and (days_to_expire
, ifexpired
flag isfalse
)
Creating Alarm Trigger for System Restart
To create an alarm trigger for a system restart, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Restart Services" input.service == "sheepdog"
This alarm will trigger to inform the users about the reboot operation on the CipherTrust Manager.
Creating Alarm Trigger for Service Restart
To create an alarm trigger for a service restart, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Restart Services"
This alarm will trigger to inform the users about a restart performed on a service.
Creating Alarm Trigger to Enable Prometheus Metrics
To create an alarm trigger to enable Prometheus metrics on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Enable Prometheus Metrics Collection" input.service == "kylo"
This alarm will trigger to inform the users that Prometheus metrics are enabled on the CipherTrust Manager.
Creating Alarm Trigger to Disable Prometheus Metrics
To create an alarm trigger to disable Prometheus metrics on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Disable Prometheus Metrics Collection" input.service == "kylo"
This alarm will trigger to inform the users that Prometheus metrics are disabled on the CipherTrust Manager.
Creating Alarm Trigger for Certificate Expiration Warning
To create an alarm trigger for a certificate expiration warning, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Following certificates are expired or going to expire in a few days. To avoid a possible interruption of your service, please renew your certificates." input.message == "Certificate Expiry Check" input.service == "naboo"
This alarm will trigger to inform the user about the certificates that are going to be expired in near future or have already expired.
Creating Alarm Trigger for Multiple Login Failure Occurrences
To create an alarm trigger for multiple login failures occurrences on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
"input.message == "error getting client/user: Wrong username or password." input.message == "Create Token"
Creating Alarm Trigger for Failed System Backup using SCP
To create an alarm trigger for failed system backup using SCP, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.details.status == "failed" input.message == "SCP Backup"
This alarm will trigger to inform the user about the failure of securely copying a backup to the set destination.
Creating Alarm Trigger for Unauthorized Key Usage
To create an alarm trigger for unauthorized key usage on the CipherTrust Manager, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "read on target is not authorized: verdict included conditions, which are not supported: ReadKey" input.service == "minerva"
This alarm will trigger to inform the user about an unauthorized attempt to access an existing key on the CipherTrust Manager.
Creating Alarm Trigger when Disk/Key Storage is Full
To create an alarm trigger when the disk/key storage on the CipherTrust Manager is full, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Disk Full" input.details.previous_state == "off" input.details.service == "Sheepdog" input.details.state == "on" input.service == "Sheepdog"
This alarm will trigger to inform the user that storage on the CipherTrust Manager has been consumed more than the configured threshold limit of 80%.
Creating Alarm Trigger for Unreachable HSM
To create an alarm trigger when HSM is offline, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "HSM is offline" input.details.name == "HSM Offline" input.details.service == "Darkstar" input.details.source == "ciphertrust" input.details.source_type == "server_record" input.service == "Darkstar" input.source == "ciphertrust"
This alarm will trigger to inform the user that the CipherTrust Manager is unable to access the HSM.
Note
To trigger this alarm, the HSM should be configured on the CipherTrust Manager as a Root of Trust.
Creating Alarm Trigger when Cluster Node is Unavailable
To create an alarm trigger when a node in a cluster goes down, follow the steps mentioned in the Configuring alarm triggers based on a record section, and specify the Condition action as:
input.message == "Cluster Node Down" input.details.service == "dbmgr" input.details.source_type == "server_record" input.service == "dbmgr"
This alarm will trigger to inform the user about the unavailability of a node within the cluster.