Deploying CCC in High Availability Configurations

This page demonstrates how to implement a sample configuration for deploying CCC in a high availability (HA) setup. CCC offers an open and flexible architecture that supports various configuration options, including high availability. This guide covers everything from optimizing your HA setup, file sharing, and validating your HA configuration to installing the server OS, configuring the network, setting up PostgreSQL servers in HA mode, and configuring the CCC application server. Follow these steps to ensure a robust and resilient CCC deployment:

• Evaluate system scalability and resilience

• Optimize your configuration for HA

• Set up file sharing

• Validate HA configuration

• Install server OS and configure network

• Configure and set up PostgreSQL server in HA mode

• Set up and configure CCC application server

Evaluate system scalability and resilience

Explore the figure below illustrating a comprehensive CCC configuration designed for high availability. Identify key components and evaluate how they contribute to the system's resilience and scalability. Compare this setup with your current infrastructure to gauge its suitability for meeting your operational needs. Consider implementation challenges and scalability aspects to ensure a robust deployment aligned with your organization's requirements.

To use the HA configuration, you need to use an external database.

Optimize your configuration for HA

By adding additional and standby components to the basic configuration, you can create an HA system that offers the following features:

Server redundancy

A true high-availability system must continue operating without interruption if one of the system components fails. To achieve this capability, it is recommended to set up the following components:

• Two worker nodes running CCC servers

• One master node for Kubernetes

• Two PostgreSQL database servers

• Two NFS servers

Load balancing

The Kubernetes master node handles load balancing by forwarding requests to all CCC servers. Each request is taken in sequence and routed to one of the worker nodes running CCC servers. If a CCC server goes down, the request is seamlessly forwarded to the next available server, with no impact on the end user. Kubernetes is configured to support persistent (or sticky) sessions, ensuring that all requests for a specific user session are directed to the same CCC server. In the event of a failover, the user will be prompted to log in to another active server, and all subsequent requests will be directed to the new server for the remainder of the session.

Data replication

It is recommended to configure the PostgreSQL database for streaming replication between the active and standby database servers. This setup ensures uninterrupted access to the data stored in the CCC database in the event of a failure. Any HA implementation should include this or a similar feature. Similarly, configure the NFS server for streaming replication of CCC data between the active and standby NFS servers. This ensures continuous access to the data stored on the NFS server in case of a failure. Any HA implementation should include this or a similar feature.

Failover protection

Kubernetes automatically redirects users to an active CCC server if one of the CCC servers on a worker node goes down. To manage database failover, you can install an application like keepalived on each database server. This allows you to use a virtual IP address to identify the active database server. If the active server goes down, the standby server assumes the virtual IP address and becomes the active server. Additionally, notification of a failover is necessary to re-synchronize the databases when the failed server is brought back online. Similarly, NFS server failover is managed by installing keepalived on each NFS server. This setup uses a virtual IP address to identify the active NFS server. If the active server goes down, the standby server takes over the virtual IP address and becomes the active server.

Operations in progress during a failover may fail.

Set up file sharing

To set up file sharing among CCC servers, follow these steps on each NFS server:

Unmount NFS directory during CCC uninstall

To unmount the NFS directory during uninstallation, the CCC administrator should run the following command as the root user:

umount -f -l /home/ccc/packages

This step should be performed only if CCC is set up in HA mode and should be executed on all CCC servers that have been set up as NFS clients.

Validate HA configuration

This section provides an overview of the HA configuration validated by the engineering team.

Master node

The Kubernetes master node serves as the control panel for the CCC cluster on Kubernetes. It manages requests, maintains desired replicas, handles load balancing, ensures failover protection, manages persistent volumes, and configures network settings for the cluster.

Worker nodes (Redundant CCC application servers)

CCC applications are deployed on two separate worker nodes. These nodes receive requests from the master node, which distributes them to individual CCC servers. In case one server becomes unavailable, requests are automatically redirected to the other available server.

PostgreSQL servers

PostgreSQL is installed on two distinct Linux workstations and configured with streaming replication. Each database server is equipped with keepalived for failover and notification mechanisms.

Install server OS and configure network

An HA deployment requires seven separate servers that exchange data over the network to operate as a unified system. When configuring CCC in HA, you need to specify the network address (IP address or hostname) of specific servers in the deployment. To simplify deployment and avoid potential misconfigurations, follow these steps:

Configure and set up PostgreSQL server in HA mode

This section describes how to set up a PostgreSQL HA cluster configuration consisting of a primary PostgreSQL server and a standby PostgreSQL server using streaming replication. Follow these steps to set up your PostgreSQL HA cluster:

• Install PostgreSQL

• Configure the primary PostgreSQL database server

• Configure the standby PostgreSQL database server

• Configure client authentication and SSL

• Set up users and configure firewall

• Test the PostgreSQL database cluster

• Set up keepalived on the PostgreSQL servers

• Test keepalived

This procedure assumes the use of PostgreSQL 14. For more detailed information, refer to the PostgreSQL documentation at PostgreSQL 14 Documentation.

Install PostgreSQL

You will need two standalone Linux servers: one for the primary server and one for the standby server. The tested and documented configuration uses CentOS 7 and PostgreSQL 14. Other operating systems may work, although they are not tested and may have different paths for some components.

Configure the primary PostgreSQL database server

To configure the primary PostgreSQL database server:

su root
umask 0022

listen_addresses = '<db_primary_IP_or_hostname>,<keepalived_virtual_IP>'
ssl = on
wal_level = hot_standby
checkpoint_segments = 32
archive_mode = on
archive_command = 'cp %p /tmp/%f'
max_wal_senders = 3
wal_keep_segments = 32

host replication all <standby_server_IP> md5
host all all <network_address/mask> md5

systemctl restart postgresql-14

sudo -u postgres psql
CREATE ROLE replication WITH REPLICATION PASSWORD '<password>' LOGIN;

sudo -u postgres pg_basebackup -h <db_primary_IP_or_hostname> -D /var/lib/pgsql/14/data -P -U replication --wal-method=stream

Configure the standby PostgreSQL database server

To configure the standby PostgreSQL database server:

su root
umask 0022

systemctl stop postgresql-14

rm -rf /var/lib/pgsql/14/data/*

rsync -av /path/to/base/backup/* /var/lib/pgsql/14/data/

standby_mode = 'on'
primary_conninfo = 'host=<db_primary_IP_or_hostname> port=5432 user=replication password=<password>'
trigger_file = '/tmp/postgresql.trigger.5432'

systemctl start postgresql-14

sudo -u postgres psql -c "SELECT * FROM pg_stat_replication;"

Configure client authentication and SSL

To configure client authentication and SSL:

host replication replicator <db_secondary_IP_or_hostname>/32 md5
hostssl lunadirectordb lunadirector <ccc1_IP_or_hostname>/32 md5
hostssl lunadirectordb lunadirector <ccc2_IP_or_hostname>/32 md5

systemctl restart postgresql-14.service

cd /var/lib/pgsql/14/data
openssl req -new -text -out server.req -nodes
openssl rsa -in privkey.pem -out server.key
rm -f privkey.pem
openssl req -x509 -in server.req -text -key server.key -out server.crt
chmod og-rwx server.key
chown postgres:postgres server.key
systemctl restart postgresql-14

Set up users and configure firewall

To set up users and configure firewall:

su - postgres -c "psql postgres postgres -c \"CREATE USER replicator REPLICATION LOGIN PASSWORD 'dbpass';\""
su - postgres -c "psql postgres postgres -c \"CREATE USER lunadirector encrypted PASSWORD 'password';\""
su - postgres -c "psql postgres postgres -c \"CREATE DATABASE lunadirectordb OWNER lunadirector;\""

iptables -I INPUT 2 -p tcp -m tcp --dport 5432 -j ACCEPT

Test the PostgreSQL database cluster

To verify that streaming replication is configured correctly, follow these steps to create a table on the primary database and confirm it is replicated on the standby database:

su - postgres -c "psql postgres postgres -c \"CREATE TABLE test (name char(10));\""

su root

systemctl start postgresql-14

su - postgres
psql -d postgres

\dt *.*

su - postgres -c "psql postgres postgres -c \"DROP TABLE test;\""

su - postgres -c "psql postgres postgres -c \"CREATE TABLE test (name char(10));\""

ERROR: cannot execute CREATE TABLE in a read-only transaction

Set up keepalived on the PostgreSQL servers

You must install the keepalived software on each PostgreSQL server to manage failover to the standby server in the event of an outage on the primary server. Once keepalived is installed and configured, if the primary server goes down, the standby server takes over as the new primary server, and the old primary server becomes the standby server. Keepalived allows you to configure a virtual IP address for the database cluster, so that database failover is transparent. For more information, refer to the keepalived documentation, available at keepalived.org. Here are the installation and configuration steps:

yum install keepalived

    ! Configuration File for keepalived

    global_defs {
        notification_email {
        }
        notification_email_from <email_address>
        smtp_server <smtp_server_IP_or_hostname>
        smtp_connect_timeout 30
        router_id CCC_DB_MONITOR
    }

    vrrp_instance VI_1 {
        state MASTER
        interface <eth0 | eth1 | eth2 | ...>
        virtual_router_id 51
        priority 101
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass PASSWORD
        }
        virtual_ipaddress {
            <db_cluster_virtual_IP> dev <eth0 | eth1 | eth2 | ...>
        }
    }

    ! Configuration File for keepalived

    global_defs {
        notification_email {
        }
        notification_email_from <email_address>
        smtp_server <smtp_server_IP_or_hostname>
        smtp_connect_timeout 30
        router_id CCC_DB_MONITOR
    }

    vrrp_instance VI_1 {
        state BACKUP
        interface <eth0 | eth1 | eth2 | ...>
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass PASSWORD
        }
        virtual_ipaddress {
            <db_cluster_virtual_IP> dev <eth0 | eth1 | eth2 | ...>
        }
        notify_master /root/<path_to_notify_master_script>
        notify_backup /root/<path_to_notify_backup_script>
    }

iptables -I INPUT -i eth2 -d 224.0.0.0/8 -j ACCEPT

systemctl start keepalived

systemctl restart postgresql-14

Test keepalived

You can verify that keepalived is working by performing the following tasks on both database servers:

ip addr show <eth0 | eth1 | eth2 | ...>

Set up and configure CCC application server

Setting up and configuring the CCC application servers involves performing the following tasks on each server used to host the CCC application. To set up and configure the CCC servers:

kubectl scale --replicas=2 deployment ccc-deployment

kubectl logs –f <pod-name>