Containerized Overcloud Control Services Backup

Solution Verified - Updated

Environment

  • Red Hat OpenStack Platform (RHOSP) 12
  • Red Hat OpenStack Platform (RHOSP) 13
  • Red Hat OpenStack Platform (RHOSP) 14
  • Red Hat OpenStack Platform (RHOSP) 15
  • Red Hat OpenStack Platform (RHOSP) 16

Issue

As noted in this Solution Article there are currently no all inclusive overcloud backup solution. However, for Red Hat OpenStack 10 to 11 this backup guide can be used to at least backup crucial control plane services which would help recovery a great deal.
Wish to have the same backup procedure for control plane service on OpenStack 12, 13 and 14.

Resolution

Important
This procedure only includes crucial control plane services. It does not include backups of Compute node workloads, data on Ceph Storage nodes, nor any additional services.

NOTE
For RHOSP13, control plane backup procedure is officially provided in Undercloud and Control plane back up and restore.

Log into a Controller node. You can access the overcloud from the undercloud. Once on the controller, sudo -i to get root user.
From here we will backup the same things as in the aforementioned guide.

MYSQL:
If you are using Openstack 12 to 15

# mkdir -p /var/tmp/mysql_backup/
# ln -s /var/lib/config-data/puppet-generated/mysql/root/.my.cnf ~/.my.cnf
# mysql -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" | xargs mysqldump -uroot --single-transaction --databases > /var/tmp/mysql_backup/openstack_databases-`date +%F`-`date +%T`.sql
# mysql -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -s -N -e | sed 's/$/;/' > /var/tmp/mysql_backup/openstack_databases_grants-`date +%F`-`date +%T`.sql

If you are using Openstack 16

mkdir -p /var/tmp/mysql_backup/
BACKUPDATE=$(date +%F)-$(date +%T)
GALERACONTAINER=$(podman ps -f name=galera --format={{.Names}})

for i in $(podman exec $GALERACONTAINER mysql -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';"); do podman exec $GALERACONTAINER /usr/bin/mysqldump -uroot --single-transaction --databases $i >> /var/tmp/mysql_backup/openstack_databases-${BACKUPDATE}.sql; done

for i in $(podman exec $GALERACONTAINER mysql -s -N -e "SELECT CONCAT('''',user,'''@''',host,'''') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')"); do podman exec $GALERACONTAINER mysql -s -N -e "SHOW GRANTS FOR $i" >> /var/tmp/mysql_backup/openstack_databases_grants-${BACKUPDATE}.sql; done

Copy the "/var/tmp/mysql_backup/" directory to a safe location.

MONGO:
If you are using Openstack 12 to 15
Do note that some overclouds will not have mongodb running any more on the overcloud.

# ss -lntp | awk '/27017.*mongo/ {print $4}'
# mkdir -p /var/tmp/mongo_backup/
# mongodump --oplog --host $MONGOIP --out /var/tmp/mongo_backup/

Copy the "/var/tmp/mongo_backup/" directory to a safe location.

REDIS:
If you are using Openstack 12 to 15

# REDISIP=$(awk '/^bind/ {print $NF}' /var/lib/config-data/puppet-generated/redis/etc/redis.conf)
# REDISPASS=$(awk '/^masterauth/ {print $NF}' /var/lib/config-data/puppet-generated/redis/etc/redis.conf)
# redis-cli -a $REDISPASS -h $REDISIP bgsave

If you are using OpenStack 16

# REDISIP=$(awk '/^bind/ {print $NF}' /var/lib/config-data/puppet-generated/redis/etc/redis.conf)
# REDISPASS=$(awk '/^masterauth/ {print $NF}' /var/lib/config-data/puppet-generated/redis/etc/redis.conf)
# REDISCONTAINER=$(podman ps -f name=redis-bundle --format={{.Names}})

# podman exec -ti $REDISCONTAINER redis-cli -a $REDISPASS -h $REDISIP bgsave

This stores the database backup in the default /var/lib/redis/ directory. Copy this database dump to a secure location.

FILES:
For all OpenStack versions covered by this solution

# mkdir -p /var/tmp/filesystem_backup/
# tar --ignore-failed-read --xattrs \
    -zcvf /var/tmp/filesystem_backup/`hostname`-filesystem-`date '+%Y-%m-%d-%H-%M-%S'`.tar \
    /etc \
    /srv/node \
    /var/log \
    /var/lib/nova \
    --exclude /var/lib/nova/instances \
    /var/lib/glance \
    /var/lib/keystone \
    /var/lib/cinder \
    /var/lib/heat \
    /var/lib/heat-config \
    /var/lib/heat-cfntools \
    /var/lib/rabbitmq \
    /var/lib/neutron \
    /var/lib/haproxy \
    /var/lib/openvswitch \
    /var/lib/redis \
    /usr/libexec/os-apply-config \
    /home/heat-admin \
    /var/lib/config-data/ \
    /var/lib/kolla/

Copy the resulting tar file to a secure location.

Root Cause

There are no official or complete backup solution for the overcloud. This work is slated for Red Hat OpenStack 16 as per this This content is not included.bugzilla.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.