Database Size Management for Red Hat Enterprise Linux OpenStack Platform

Updated

Red Hat Enterprise Linux OpenStack Platform uses a MariaDB database in the OpenStack control plane for data persistence. This database acts as a system backend, storing configuration and historical data for certain OpenStack services, including Compute, Identity Service, and Block Storage. This article discusses recommended practices for maintaining the size of the database with regards to automatic purging of unneeded data.

This document will refer to each service of OpenStack individually.

Identity Service (Keystone)

The Keystone service grants access to applications using expirable security tokens that are generated on a per-use basis. The tokens expire typically within a scope of hours or even less, however remain present in the Keystone database as rows in the token table until purged. Over time, the size of the token table will continue to grow unbounded if not cleaned.

Management of the token table is performed by the keystone-manage command, from the command line this looks like:

/usr/bin/keystone-manage token_flush

In order to automate this command, it should run within the scope of a cron job on at least an hourly basis. As of Red Hat Enterprise Linux OpenStack Platform 4, the Puppet installer will establish this cron automatically. For older versions of Red Hat Enterprise Linux OpenStack Platform, or if for some reason the cron needs to be manually installed, a cron entry to run this command on an hourly basis looks like:

0 */1 * * * /usr/bin/keystone-manage token_flush >/dev/null 2>&1

It is also worth noting that older versions of Keystone’s management tool (pre-Juno) have an issue where a very large token table (e.g. hundreds of thousands of rows) may fail to be flushed; the reason for this is that the command attempts to run a DELETE across the entire table at once, producing a transaction that is too large for Galera to process based on the default settings of 128K rows for wsrep_max_ws_rows and 1GB for wsrep_max_ws_size. Having the cron run at least hourly should ensure the table size stays very small. In modern Keystone versions, the flush operation is batched in groups of 1000 rows at a time when the MySQL backend is in use.

Compute Service (Nova)

Nova Compute makes use of a database concept known as a “soft-deleted” row. This is a database row that for all intents and purposes is considered to be a “deleted” record, however a SQL DELETE command is not actually emitted; instead, a status column in the table is set to denote this row as “deleted”, so that the application considers it as such, however the row remains in the table.

As a result of this design, those tables in Nova that make use of “soft delete” are subject to large amounts of growth, consisting of rows that are not accessed in any way and are only useful for auditing purposes. Therefore it’s necessary that these rows be periodically purged.

The Nova application includes a command nova-manage db archive_deleted_rows which is intended to achieve this purpose. However, in recent versions of Nova this command has been demonstrated to not function completely (see https://bugs.launchpad.net/nova/+bug/1183523 ). It is safe to install as a cron, as below where we set it up to run every twelve hours:

0 */12 * * * /usr/bin/nova-manage db archive_deleted_rows >/dev/null 2>&1

The Nova project is in the process of working to improve upon this command by building out a comprehensive system of purging soft deleted records (see https://review.openstack.org/#/c/200224/8 for the current specification of this). There are also proposals to cease making use of soft-deleted records altogether (see https://review.openstack.org/#/c/184645/).

Block Storage (Cinder)

The Cinder application, like Nova, also makes use of “soft-deleted” rows. Cinder as of the Kilo release includes a utility for deleting these rows, given a number of days denoting the minimum age of the rows to delete. A crontab entry to run this nightly, removing rows deleted as of one day, is as follows:

0 */24 * * * /usr/bin/cinder-manage db purge 1 >/dev/null 2>&1

For older versions of Cinder, the soft deleted rows can only be removed using manual SQL commands or custom scripting.

Image Service (Glance)

Glance also makes use of “soft-deleted” rows, however currently these can only be removed using manual SQL commands or custom scripting. The blueprint at https://blueprints.launchpad.net/glance/+spec/database-purge refers to a specification to add a “db purge” command to Glance.

Networking Service (Neutron)

Neutron currently does not make use of soft-deleted rows nor does it have tables that grow unbounded.

Product(s)
Category
Article Type