Integrating CTE with a Kafka

This document describes how to integrate CTE with Apache Kafka.

Test Environment

• CTE Agent: 7.5.0 and 7.6.0

• CipherTrust Manager: 2.15 & 2.16

• OS: Ubuntu 22.04.02 and RHEL 9.3

• Kafka version: Kafka 2.13-3.7.0

• Java version: openjdk-11-jdk

• File System: XFS and EXT4

To integrate CTE with a Kafka, install configure Kafka, install and register the CTE Agent, configure the Settings.

Install and Register CTE Agent

• Install CTE-LDT v7.6, or a subsequent version, enable and Register it with CipherTrust Manager.

Install Kafka

Download and extract the Kafka binaries into dedicated folders in your Kafka user’s home directory from one of the following URLs:

1. Download the Kafka binaries into dedicated folders in your Kafka user’s home directory, type:

   $ wget https://downloads.apache.org/kafka/3.4.0/kafka_2.12-3.4.0.tgz
   

2. Unzip the file, type:

   $ tar -xzf kafka_2.12-3.4.0.tgz
   

3. Rename the file, type:

   $ mv kafka_2.12-3.4.0  kafka
   

Creating systemd unit files to start the Kafka server

1. Display the contents of the zookeeper.service file:

   $ cat /etc/systemd/system/zookeeper.service
   

   Response

   [Unit]
   Requires=network.target remote-fs.target
   After=network.target remote-fs.target
   
   [Service]
   Type=simple
   User=kafka
   ExecStart=/root/kafka/kafka/bin/zookeeper-server-start.sh /root/kafka/config/zookeeper.properties
   ExecStop=/root/kafka/bin/zookeeper-server-stop.sh
   Restart=on-abnormal
   
   [Install]
   WantedBy=multi-user.target
   

2. Display the contents of the kafka.service file:

   $ cat /etc/systemd/system/kafka.service
   

   Response

   [Unit]
   Requires=zookeeper.service
   After=zookeeper.service
   
   [Service]
   Type=simple
   User=kafka
   ExecStart=/bin/sh -c '/root/kafka/bin/kafka-server-start.sh /root/kafka/config/server.properties > /root/kafka/kafka.log 2>&1'
   ExecStop=/root/kafka/bin/kafka-server-stop.sh
   Restart=on-abnormal
   
   [Install]
   WantedBy=multi-user.target
   

3. Enable Zookeeper, type:

   $ systemctl enable zookeeper
   

4. Check the status of Zookeeper, type:

   $ systemctl status zookeeper
   

5. Enable Kafka, type:

   $ systemctl enable kafka
   

6. Start Kafka, type:

   $ systemctl start kafka
   

7. Add the kafka.service and zookeeper.service to the following file: /lib/systemd/system/secfs-fs-barrier.service

Using Kafka with CTE

1. Make sure that the Kafka service is not running, type:

   systemctl stop kafka
   

2. Create a wide Open LDT policy.

   Security Rule

   Order	Action	Effect	Browsing
   1	key_op	permit,applykey	No
   2	all_ops	permit,audit,applykey	Yes
   3	all_ops	permit,audit	Yes

   Key Rule

   Order	Current Key Name	Transformation Key Name	Exclusion Rule
   1	clear_key	ldt-kafka-key	No

3. Guard the /kafka-logs/ custom directory.

   GuardPoint	Policy	Type	ConfigState	Status
   /kafka-logs	LDT-Kafka-Policy	local	guarded	guarded

4. Start the Kafka service, type:

   systemctl start kafka
   

Verifying Data Encryption

1. Verify that LDT encrypted the Kafka logs, type:

   voradmin ldt attr get /kafka-logs/recovery-point-offset-checkpoint
   

   Response

   LDT attributes: rekeyed_size=0, rekey_status=none
   Key:    name=ldt-kafka-key, version=2
   

2. Unguard the Kafka Logs Directory.