fence_scsi stonith device fails to start or function in a RHEL 7 Update 1 High Availability cluster and logs show "Failed: nodename or key is required"

Environment

• Red Hat Enterprise Linux (RHEL) 7 with the High Availability Add On
• Releases of fence-agents-scsi starting with 4.0.11-11.el7 and prior to 4.0.11-27.el7
• One or more stonith devices configured to use agent fence_scsi

Issue

• fence_scsi fails to work after updating to RHEL 7.1
• After setting up a fence_scsi stonith device, pcs status shows errors for that device on all nodes and the device is listed as "Stopped"

Failed actions:
    scsistonith_start_0 on rhel72.example.com 'unknown error' (1): call=37, status=Error, exit-reason='none', last-rc-change='Sun Apr 19 16:37:03 2015', queued=0ms, exec=1163ms
    scsistonith_start_0 on rhel71.example.com 'unknown error' (1): call=14, status=Error, exit-reason='none', last-rc-change='Sun Apr 19 16:38:52 2015', queued=0ms, exec=1129ms

• There are "Failed: nodename or key is required" errors in my logs for my fence_scsi stonith device

Apr 19 16:38:52 rhel71 crmd[1549]: notice: process_lrm_event: Operation scsistonith_monitor_0: not running (node=rhel71.example.com, call=13, rc=7, cib-update=14, confirmed=true)
Apr 19 16:38:52 rhel71 stonith-ng[1545]: notice: stonith_device_register: Added 'scsistonith' to the device list (2 active devices)
Apr 19 16:38:52 rhel71 fence_scsi: Failed: nodename or key is required
Apr 19 16:38:52 rhel71 fence_scsi: Please use '-h' for usage
Apr 19 16:38:54 rhel71 fence_scsi: Failed: nodename or key is required
Apr 19 16:38:54 rhel71 fence_scsi: Please use '-h' for usage
Apr 19 16:38:54 rhel71 stonith-ng[1545]: notice: log_operation: Operation 'monitor' [2525] for device 'scsistonith' returned: -201 (Generic Pacemaker error)
Apr 19 16:38:54 rhel71 stonith-ng[1545]: warning: log_operation: scsistonith:2525 [ Failed: nodename or key is required ]
Apr 19 16:38:54 rhel71 stonith-ng[1545]: warning: log_operation: scsistonith:2525 [  ]
Apr 19 16:38:54 rhel71 stonith-ng[1545]: warning: log_operation: scsistonith:2525 [ Please use '-h' for usage ]
Apr 19 16:38:54 rhel71 stonith-ng[1545]: warning: log_operation: scsistonith:2525 [  ]
Apr 19 16:38:54 rhel71 crmd[1549]: error: process_lrm_event: Operation scsistonith_start_0 (node=rhel71.example.com, call=14, status=4, cib-update=15, confirmed=true) Error

Resolution

• Update to fence-agents-4.0.11-27.el7 or later

• Workaround: Specify pcmk_monitor_action=metadata as an attribute on the stonith device in addition to the This content is not included.usual attributes for fence_scsi. For example, when creating the stonith device:

  # pcs stonith create scsi fence_scsi pcmk_host_list="node1.example.com node2.example.com" pcmk_monitor_action="metadata" pcmk_reboot_action="off" devices="/dev/mapper/mpatha" meta provides="unfencing"
  

  Or to update an existing device:

  # pcs stonith update scsi pcmk_monitor_action="metadata"
  

  The stonith device may need cleaning up after making such an update:

  # pcs stonith cleanup scsi
  

Root Cause

Red Hat corrected this issue via Bugzilla # #1213571.

In RHEL 7 Update 1, fence_scsi was updated to a new implementation with many improvements and is written in python to utilize the shared fencing library. The old implementation did not support a monitor action, and when stonithd would call the agent with this action, it would simply print the usage information and exit; however that exit would be clean, as the return code would be zero. The newer implementation also does not support a monitor action, but it fails explicitly because no nodename or key attribute is supplied (stonithd doesn't pass in nodename to the monitor action like it does for other actions, since Content from fedorahosted.org is not included.monitor is not checking the status of any particular node).

Overriding pcmk_monitor_action to point to metadata causes this action to always succeed, which will bypass the problem.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.