Data Management
This section desribes the processes and strategies that are used for collecting, storing, organizing, protecting, and maintaining data in CCC, with particular focus on persistence, backup, and restore mechanisms. Persistence ensures that data is not lost when the CCC container is shut down, while backup and restore are the key components of data protection and disaster recovery strategies. These processes contribute towards ensuring the availability, integrity, and security of CCC data, and form the core of CCC data management.
Persistence
The persistence mechanisms supported by CCC include bind mount and volume mount.
Bind Mount
The default persistence mechanism used by CCC is bind mount, wherein the CCC container mounts a file or directory from the host machine's file system into the container's file system. This allows the container to access and modify the data in the file or directory as if it were part of its own file system. Any changes made to the data by the container are also reflected on the host machine's file system, thus persisting the changes even after the container is stopped or deleted.
Podman
For users who have installed CCC with Podman, the /var/lib/postgresql directory of the CCC container is mapped to the ccc_dist_folder/podman/pgdata directory on the host machine. The entire CCC database is stored inside the pgdata directory. When the podman-compose up command is used to initialize the CCC container, the container obtains the mapping from the podman-compose.yml file and begins persisting data according to the defined specifications.
You can change the directory that has been mapped on the host machine by modifying the podman-compose.yml file.
Kubernetes
For users who have installed CCC with Kubernetes, the /var/lib/postgresql directory of the CCC container is mapped to /home/ccc/pgdata directory on the host machine. The entire CCC database is stored inside the pgdata directory. When the CCC container is initialized, it obtains the mapping from the postgres-data.yaml file and begins persisting data according to the defined specifications.
You can change the directory that has been mapped on the host machine by modifying the postgres-data.yaml file.
Volume Mount
To change the CCC persistence mechanism to volume mount that enables data to be persisted and shared between containers, or between a container and the host machine, you need to make the following changes, depending on the orchestration mechanism that has been used while installing CCC:
Podman
If you’ve installed CCC using Podman and want to change the persistence mechanism to volume mount, use the steps outlined below:
Open the podman-compose.yml file in the podman directory.
Go to the volumes section.
Replace -./pgdata:/var/lib/postgresql with -ccc_pgdata:/var/lib/postgresql:Z,U.
Add a new Volumes section in podman-compose.yml file, with the following details:
volumes:
ccc_pgdata:
Ensure that the updated podman-compose.yml file appears as follows after you've made these changes:
The Podman volume data gets stored at:
/home/user/.local/share/containers/storage/volumes/podman_volume.
Therefore, after changing the persistence mechanism to volume, the data directory will be at:
/home/user/.local/share/containers/storage/volumes/podman_ccc_pgdata.
Kubernetes
If you’ve installed CCC using Kubernetes and want to change the persistence mechanism to volume mount, use the steps outlined below:
Open the deployment.yaml file in the Kubernetes directory.
Scroll to the volumes section.
Replace the existing content of the volumes section with the following content:
- name: lunalogs
emptyDir: {}
- name: packages
emptyDir: {}
- name: ccclogs
emptyDir: {}
- name: pgdata
emptyDir: {}
- name: certificates
hostPath:
path: /home/ccc/ccc-certs/
type: Directory
It is not advisable to use the volume mount mechanism to persist data in Kubernetes. Kubernetes stores data at the path /var/lib/kubelet/pods/pod-id/volumes/kubernetes.io~empty-dir inside the worker node and if the pod or deployment gets deleted, any data that is persisted using this approach will be lost permanently.
Backup and Restore
The approach you need to use to backup and restore CCC data depends on whether you are using an internal database or an external database.
Internal Database: Backup and Restore
External Database: Backup and Restore
Internal Database: Backup and Restore
If CCC is using an internal database, it utilizes the postgresql database running within the same container. To ensure data persistence, a bind mount mechanism is employed, whereby the /var/lib/postgresql directory of the CCC container is linked to the host machine. To backup and restore data, you can use either of the following two methods:
Method 1: Backup and restore using pg_dump
Method 2: Backup and restore using pgdata
Method 1: Backup and restore using pg_dump
This method involves creating a pg_dump backup file and then restoring it.
Backup
The steps involved in performing a backup are as follows:
Create a pg_dump file by executing below command:
Podman
podman exec ccc bash -c "/usr/pgsql-14/bin/pg_dump 'host=localhost port=5432 dbname=lunadirectordb user=lunadirector password=temp1234' > postgres.sql"
Kubernetes
kubectl exec -it <podname> -- bash -c "/usr/pgsql-14/bin/pg_dump 'host=localhost port=5432 dbname=lunadirectordb user=lunadirector password=temp1234' > postgres.sql"
Copy the pg_dump file from container to host machine to make it available for future.
Podman
podman cp ccc:/usr/safenet/ccc/postgres.sql .
Kubernetes
kubectl cp <podname>:/usr/safenet/ccc/postgres.sql .
Restore
This method involves restoring the pg_dump backup file. The steps involved are as follows:
Copy the backup pg_dump file into the container directory /usr/safenet/ccc/.
Podman
podman cp postgres.sql ccc:/usr/safenet/ccc/
Kubernetes
kubectl cp postgres.sql <podname>:/usr/safenet/ccc/
Execute the below command to restore the backup data:
Podman
podman exec ccc bash -c "cd scripts && sh dbmig.sh '<ccc_admin_password>'"
Kubernetes
kubectl exec <podname> bash -c "cd scripts && sh dbmig.sh '<ccc_admin_password>'"
If the CCC admin password contains special characters, we need to enter those with escape. For example, if the CCC admin password is "1q@W3e$R", we need to enter it as "1q\@W3e\$R".
The name of the pg_dump file should be "postgres.sql".
Method 2: Backup and Restore using pgdata
This method involves making a backup of pgdata directory at a safe location and restoring it back.
Backup
To create a backup, copy the pgdata directory and place it at a safe location as a backup.
Restore
To restore the database from the backup file, perform these steps:
Go to the podman directory where the podman-compose.yml file is present and remove the running CCC container by using the following command:
podman-compose down
Remove the current pgdata directory in the podman directory.
Copy the backup pgdata directory and paste it into the podman directory.
Go to podman directory, update the ccc_config.env file if needed, and then execute the following command:
podman-compose up -d
External Database: Backup and Restore
If you are utilizing an external database and need to perform data backup and restore, refer to the procedure outlined on the official postgres website:
https://www.postgresql.org/docs/current/backup.html
