Using CTE with Apache Kafka

This document describes how to integrate CTE with Apache Kafka.

Test Environment

• CTE Agent: 7.5.0 and 7.6.0

• CipherTrust Manager: 2.15 & 2.16

• OS: Ubuntu 22.04.02 and RHEL 9.3

• Kafka version: Kafka 2.13-3.7.0

• Java version: openjdk-11-jdk

• File System: XFS and EXT4

To integrate CTE with a Kafka, install configure Kafka, install and register the CTE Agent, configure the Settings.

Install and Register CTE Agent

• Install CTE-LDT v7.6, or a subsequent version, enable and Register it with CipherTrust Manager.

Install Kafka

Download and extract the Kafka binaries into dedicated folders in your Kafka user’s home directory from one of the following URLs:

1. Download the Kafka binaries into dedicated folders in your Kafka user’s home directory, type:

   $ wget https://downloads.apache.org/kafka/3.4.0/kafka_2.12-3.4.0.tgz

   Copy

2. Unzip the file, type:

   $ tar -xzf kafka_2.12-3.4.0.tgz

   Copy

3. Rename the file, type:

   $ mv kafka_2.12-3.4.0  kafka

   Copy

Creating systemd unit files to start the Kafka server

1. Display the contents of the zookeeper.service file:

   $ cat /etc/systemd/system/zookeeper.service

   Copy

   Response

   [Unit]
   Requires=network.target remote-fs.target
   After=network.target remote-fs.target
   
   [Service]
   Type=simple
   User=kafka
   ExecStart=/root/kafka/kafka/bin/zookeeper-server-start.sh /root/kafka/config/zookeeper.properties
   ExecStop=/root/kafka/bin/zookeeper-server-stop.sh
   Restart=on-abnormal
   
   [Install]
   WantedBy=multi-user.target

   Copy

2. Display the contents of the kafka.service file:

   $ cat /etc/systemd/system/kafka.service

   Copy

   Response

   [Unit]
   Requires=zookeeper.service
   After=zookeeper.service
   
   [Service]
   Type=simple
   User=kafka
   ExecStart=/bin/sh -c '/root/kafka/bin/kafka-server-start.sh /root/kafka/config/server.properties > /root/kafka/kafka.log 2>&1'
   ExecStop=/root/kafka/bin/kafka-server-stop.sh
   Restart=on-abnormal
   
   [Install]
   WantedBy=multi-user.target

   Copy

3. Enable Zookeeper, type:

   $ systemctl enable zookeeper

   Copy

4. Check the status of Zookeeper, type:

   $ systemctl status zookeeper

   Copy

5. Enable Kafka, type:

   $ systemctl enable kafka

   Copy

6. Start Kafka, type:

   $ systemctl start kafka

   Copy

7. Add the kafka.service and zookeeper.service to the following file: /lib/systemd/system/secfs-fs-barrier.service

Using Kafka with CTE

1. Make sure that the Kafka service is not running, type:

   systemctl stop kafka

   Copy

2. Create a wide Open LDT policy.

   Security Rule

   Order	Action	Effect	Browsing
   1	key_op	permit,applykey	No
   2	all_ops	permit,audit,applykey	Yes
   3	all_ops	permit,audit	Yes

   Key Rule

   Order	Current Key Name	Transformation Key Name	Exclusion Rule
   1	clear_key	ldt-kafka-key	No

3. Guard the /kafka-logs/ custom directory.

   GuardPoint	Policy	Type	ConfigState	Status
   /kafka-logs	LDT-Kafka-Policy	local	guarded	guarded

4. Start the Kafka service, type:

   systemctl start kafka

   Copy

Verifying Data Encryption

1. Verify that LDT encrypted the Kafka logs, type:

   voradmin ldt attr get /kafka-logs/recovery-point-offset-checkpoint

   Copy

   Response

   LDT attributes: rekeyed_size=0, rekey_status=none
   Key:    name=ldt-kafka-key, version=2

   Copy

2. Unguard the Kafka Logs Directory.