Is there a way to store secrets in pacemaker?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux Server 8 (with the High Availability Add On)
  • Red Hat Enterprise Linux Server 9 (with the High Availability Add On)

Issue

  • Is there a way to store secrets in pacemaker?
  • Is there a way to hide passwords for stonith devices (fence-agents) or resources in the pacemaker configuration file cib.xml.

Resolution

Red Hat Enterprise Linux 8
  • The issue (bz1793860) has been resolved with errata RHEA-2020:4804 with the following package(s): pacemaker-2.0.4-6.el8 or later.

With the errata above a new feature was added to store secrets outside of pacemaker configuration file /var/lib/pacemaker/cib/cib.xml. The cibsecret utility can be used on stonith or resources configured in the cib file.

There currently is no pcs option to hide secrets, but the following bugzilla was opened to add support for this: Bug 1803995 - [RFE] pcs interface for pacemaker CIB secrets

Know Limitations

There are a couple of known limitations when using `cibsecret`:
  • This command cibsecret can only be run from full cluster nodes (not pacemaker remote nodes). If there are remote nodes that are part of cluster then location constraints should be used to ban the remote nodes from running any resource that uses cibsecret.
  • Remote nodes cannot use cibsecret with the corresponding stonith (fence-agent) device.
    .ban affected resources from all remote nodes
  • Changes are not atomic, so the cluster may use different values while a change is in progress. To avoid problems, it is recommended to put the cluster in maintenance mode when making changes with this command.
  • Changes in secret values do not trigger a reload or restart of the affected resource, since they do not change the CIB. If a response is desired before the next cluster recheck interval, any CIB change (such as setting a node attribute) will trigger it.
  • If any node is down when changes to secrets are made, or a new node is later added to the cluster, it may have different values when it joins the cluster, before cibsecret sync is run. To avoid this, it is recommended to run the sync command (from another node) before starting pacemaker on the node that was down or is a new cluster node that was added.
  • The password will still be stored in a clear-text file, protected by root-only file permissions. The main benefits are that the CIB can be given wide read access with little concern, and this capability can be used with any resource agent or fence agent (stonith device).
  • cibsecret will take advantage of fping, pssh, or pdsh if installed, but none of those are supported in RHEL. fping does greatly speed up synchronization of secrets, while parallel ssh has a smaller impact. In any case, ssh must be installed on all cluster nodes, and if it's not passwordless be prepared to enter the password a whole lot of times when running cibsecret.

Adding an attribute as a secret value to a stonith device or resource works in the same manner.

Adding an attribute value for a resource as a secret.

Create a resource. Do not include any attribute that will be set via cibsecret. If you do set an attribute when the stonith device or resource is created that will later be stored as a cibsecret then do note that only the current cib file will be updated to be a secret. The attribute value could still exist in plaintext within any of the archived cib files.

# pcs resource create d1 ocf:heartbeat:Dummy

Put cluster into maintenance-mode to avoid different attributes being present across nodes while modifying attributes .

#  pcs property set maintenance-mode=true

Store an attribute of that resource as a secret. The cibsecret command will update the cib with the attribute if it does not exist in the cib already. 

# cibsecret set d1 fake 20
INFO: syncing /var/lib/pacemaker/lrm/secrets/d1/fake to  rhel8-2.examplerh.com  ...
Set 'd1' option: id=d1-instance_attributes-fake set=d1-instance_attributes name=fake value=lrm://

The value of the attribute is now stored outside of the cib file under the /var/lib/pacemaker/lrm/secrets/<resource-id>/<parameter-name> file.

# pcs resource show d1
Warning: This command is deprecated and will be removed. Please use 'pcs resource config' instead.
 Resource: d1 (class=ocf provider=heartbeat type=Dummy)
  Attributes: fake=lrm://  <---- The "fake" attribute now reports "lrm://"
  Operations: migrate_from interval=0s timeout=20s (d1-migrate_from-interval-0s)
              migrate_to interval=0s timeout=20s (d1-migrate_to-interval-0s)
              monitor interval=10s timeout=20s (d1-monitor-interval-10s)
              reload interval=0s timeout=20s (d1-reload-interval-0s)
              start interval=0s timeout=20s (d1-start-interval-0s)
              stop interval=0s timeout=20s (d1-stop-interval-0s)

# cat /var/lib/pacemaker/lrm/secrets/d1/fake
20

Take cluster out of maintenance-mode.

#  pcs property set maintenance-mode=false

Root Cause

There is no way to store secrets or hide passwords in the cib file (/var/lib/pacemaker/cib/cib.xml) used by pacemaker. The only way to hide passwords for stonith devices defined in the cib was to create a script that contained the password.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.