Log Forwarding

Users who belong to the System Admins group can forward server and client audit records and KMIP and NAE activity logs to Elasticsearch, Loki, or a syslog server.

Elasticsearch and Loki are part of logging stacks — Elastic Stack or Grafana Loki — which provide powerful tools for querying, analyzing and visualizing CipherTrust Manager log entries. See Elastic Stack documentation and Grafana Loki documentation for full details on logging system operations and capabilities.

CipherTrust Manager always stores logs locally in addition to forwarding to configured log forwarders.

In a clustered environment, log forwarder configuration is replicated across the cluster. The currently active node sends log forwarder messages. This means that if you perform a logged operation on a node, that node sends the log record to the log forwarder.

High-Level Workflow

To configure CipherTrust Manager to forward to a logging system, there are two phases.

1. Create a log forwarder connection in Connection Manager. This establishes communication between CipherTrust Manager and the external logging system.

   Note

   CipherTrust Manager can have a total of 64 log forwarder connections. Each Elasticsearch, Loki, and Syslog connection is counted towards the 64 connection total.

2. Create a log forwarder resource on CipherTrust Manager. This object controls additional CipherTrust Manager-specific settings for the logs and records before sending them to the external logging system.

   • Configuring Elasticsearch Log Forwarder

     Controls which index names are applied to the different CipherTrust Manager log and record types.

   • Configuring Loki Log Forwarder

     Controls which labels are applied to the different CipherTrust Manager log and record types.

   • Configuring Syslog Log Forwarder

     Controls which CipherTrust Manager log and record types are forwarded to syslog.

Timezone Configuration

CipherTrust Manager server audit records and client audit records are always recorded in UTC time zone, in keeping with RFC 3339. This is important to note when you configure any external logging system such as a log forwarder or legacy syslog connection.

Log Forwarder Redirection

Log forwarder redirection is a setting to send audit records, NAE activity logs, and KMIP activity logs from a child domain to the log forwarder configured for a higher-level domain.

When an audit record, NAE activity log, or KMIP activity log is created in a child domain with log forwarder redirection, CipherTrust Manager checks the domain's immediate parent for a configured log forwarder. If the immediate parent also has log redirection enabled, CipherTrust Manager also checks its parent, the child domain's grandparent, for a configured log forwarder. CipherTrust Manager continues checking for log forwarders further up the domain hierarchy every time it finds the log redirection setting. Every time CipherTrust Manager finds a configured log forwarder in a domain, it forwards log messages to that log forwarder.

Users in the Domain Admins or Application Administrators group, such as admin can enable or disable log forwarder redirection. When a new child domain is created, log forwarder redirection is enabled by default. Existing child domains which were created in a version lower than 2.16.0 have log forwarder redirection disabled after upgrade.

Enabling or disabling log forwarder redirection in web console UI:

1. Login to the child domain as a user in the Domain Admins or Application Administrators group, such as admin.

2. Navigate to Admin Settings > Logs.

3. Click the Log Forwarders tab.

4. Click the Redirect Log Forwarder toggle to enable or disable log forwarder redirection.

Enabling or disabling log forwarder redirection in ksctl CLI:

Login to the child domain as a user in the Domain Admins or Application Administrators group, such as admin. For example:

ksctl login --user <user_with_domain_admin_permission> --password <user_password> --domain <domain_name_or_id>

• The following command disables log forwarder redirection.

  ksctl domains log-forwarders-redirection disable
  

• The following command enables log forwarder redirection.

  ksctl domains log-forwarders-redirection enable
  

• The following command displays log forwarder redirection status

  ksctl domains log-forwarders-redirection status
  

Configuring Elasticsearch Log Forwarder

The CipherTrust Manager log forwarder is compatible with Elasticsearch version 7 and 8.

You can add an index name to KMIP activity logs, NAE activity logs, server audit records, and client audit records to help with queries in the Elasticsearch environment.

To add an Elasticsearch connection you need to provide the following values:

• a connection ID of the Elasticsearch connection manager (refer to Connection Manager for details)

• a connection name for the log forwarder configuration

You can optionally provide:

• an index name for KMIP activity logs

• an index name for NAE activity logs

• an index name for server audit records

• an index name for client audit records

Syntax for Elasticsearch

ksctl log-forwarders add elasticsearch --name <name of log forwarder> --connection-id <ES connectionID/Name> --index-activity-kmip <kmip_index_name> --index-activity-nae <nae_index_name> --index-server-audit-records <server_audit_records_index_name> --index-client-audit-records <client_audit_records_index_name>

Configuring Loki Log Forwarder

You can add labels to KMIP activity logs, NAE activity logs, server audit records, and client audit records to help with queries in the Loki Grafana environment.

To add a Loki log forwarder, you must provide the following values:

• a connection ID of the Loki connection manager (refer to Connection Manager for details)

• a connection name for the log forwarder configuration

You can optionally provide:

• labels field for KMIP activity logs

• labels field for NAE activity logs

• labels field for server audit records

• labels field for client audit records

Syntax for Loki

ksctl log-forwarders add loki --name <name of log forwarder>  --connection-id <Loki ConnectionID/Name> --labels-activity-kmip <kmip_label> --labels-activity-nae <nae_label> --labels-server-audit-records <server_audit_records_label> --labels-client-audit-records <client_audit_records_label>

Configuring Syslog Log Forwarder

Once you have added a syslog connection, you can create a syslog log forwarder on CipherTrust Manager to forward KMIP activity logs, NAE activity logs, server audit records, and client audit records to Syslog server.

To add a Syslog log forwarder, you must provide:

• a connection ID of the Syslog connection manager (refer to Connection Manager for details)

• a connection name for the log forwarder configuration

You can optionally activate/deactivate:

• forward logs for activity kmip

• forward logs for activity nae

• forward logs for client audit records

• forward logs for server audit records

Syntax for Syslog

ksctl log-forwarders add syslog --name <name of log forwarder> --connection-id <Syslog ConnectionID/Name> --forward-client-audit-records <true/false> --forward-logs-activity-kmip <true/false> --forward-logs-activity-nae <true/false> --forward-server-audit-records <true/false>]

Viewing Log Forwarders

You can use ksctl log-forwarders get --id <log-forwarder-identifier> to view details for a particular log forwarder.

You can use ksctl log-forwarders list to view details for all log forwarders.

The returned details include ID, name, type (Loki or Elasticsearch), CipherTrust Manager user account, hostname, port, ElasticSearch indicies, and Loki labels.

Updating Elasticsearch Log Forwarder

For Elasticsearch log forwarder, you can modify:

• a name for the log forwarder configuration

• a connection ID of the Elasticsearch connection manager

• an index name for KMIP activity logs

• an index name for NAE activity logs

• an index name for server audit records

• an index name for server client records

Syntax for Updating Elasticsearch Log Forwarder

ksctl log-forwarders modify elasticsearch --id <LogForwarder ID/Name> --name <name of log forwarder> --connection-id <ES connectionID/Name>  --index-activity-kmip <kmip_index_name> --index-activity-nae <nae_index_name> --index-server-audit-records <server_audit_records_index_name> --index-client-audit-records <client_audit_records_index_name>

Updating Loki Log Forwarder

For Loki log forwarder, you can modify:

• a connection name for the log forwarder configuration

• a connection ID of the Loki connection manager

• Labels field for KMIP activity logs

• Labels field for NAE activity logs

• Labels field for server audit records

• Labels field for client audit records

Syntax for Updating Loki Log Forwarder

ksctl log-forwarders modify loki --id <LogForwarder ID/Name>  --name <name of log forwarder> --connection-id <Loki ConnectionID/Name> --labels-activity-kmip <kmip_label> --labels-activity-nae <nae_label> --labels-server-audit-records <server_audit_records_label> --labels-client-audit-records <client_audit_records_label>

Monitoring of Log Forwarder Connections

The CipherTrust Manager monitors the status of log forwarder connections at regular time intervals. If any connection is detected to be in an unhealthy state, an alarm is raised.

For the connections configured in the domain, the alarms are raised in that domain.

If an unhealthy log forwarder connection becomes healthy after some time, the alarm disappears for that connection. If there are multiple connection failures, the alarm will only disappear when all the connections will return to a healthy state.

When you try to clear the alarm, an error "Not Allowed to clear the log forwarder connections health check alarm" is returned.

For a cluster, alarms are node-specific because connections may fail at one node but not at another.

In the above screenshot, you can see that each alarm contains the id and name of the unhealthy log forwarder connection.

Let's consider a scenario where one more log forwarder connection "es-conn2" is added in the same domain where the "es-conn" connection already exists. Eventually, the new connection "es-conn2" also becomes unhealthy and the alarm is already raised for the existing unhealthy log forwarder connection "es-conn" as shown in the above screenshot.

In such scenarios, the alarm details will be updated only when the next cycle of monitoring completes as shown below:

Updating Syslog Log Forwarder

For Syslog log forwarder, you can modify:

• a connection name for the log forwarder configuration

• a connection ID of the Syslog connection manager

• forward logs for KMIP activity logs

• forward logs for NAE activity logs

• forward logs for server audit records

• forward logs for client audit records

Syntax for Updating Syslog Log Forwarder

ksctl log-forwarders modify syslog --id <LogForwarder ID/Name> --name <name of log forwarder> --connection-id <Syslog ConnectionID/Name> --forward-client-audit-records <true/false> --forward-logs-activity-kmip <true/false> --forward-logs-activity-nae <true/false> --forward-server-audit-records <true/false>

Deleting a Log Forwarder

To delete a log forwarder, use ksctl log-forwarders delete --id <log-forwarder-unique-identifier>.