The monitor or start operation of an oraasm resource returns a false success in a Pacemaker cluster

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux 7 or 8 (with the High Availability Add-on)
  • An ocf:heartbeat:oraasm resource

Issue

  • An ocf:heartbeat:oraasm resource is marked as "Started" in pcs status output before it's fully started.
  • Pacemaker thinks an oraasm resource is started, but the asm_smon_+ASM process or other key components are not running.
  • An oracle resource fails to start after an oraasm resource is marked "Started".
Dec 19 12:44:35 node2 crmd[6925]:  notice: Initiating start operation asm_start_0 locally on node2
Dec 19 12:44:35 node2 oraasm(asm)[33615]: INFO: Oracle ASM is not running
Dec 19 12:44:35 node2 root: Oracle HA daemon is enabled for autostart.
Dec 19 12:44:35 node2 oracle: exec /app/oragrid/12203/perl/bin/perl -I/app/oragrid/12203/perl/lib /app/oragrid/12203/bin/crswrapexece.pl /app/oragrid/12203/crs/install/s_crsconfig_pcs-hdmdb-02_env.txt /app/oragrid/12203/bin/ohasd.bin "reboot"
Dec 19 12:44:36 node2 oraasm(asm)[33615]: INFO: Oracle ASM is not running
Dec 19 12:44:37 node2 oraasm(asm)[33615]: INFO: Oracle ASM is not running
...
Dec 19 12:44:43 node2 oraasm(asm)[33615]: INFO: Oracle ASM is not running
Dec 19 12:44:44 node2 init.ohasd: bash: /sys/fs/cgroup/system.slice/oracle-ohasd.service/tasks: No such file or directory
Dec 19 12:44:44 node2 init.ohasd: bash: /sys/fs/cgroup/cpu,cpuacct/tasks: Permission denied
Dec 19 12:44:44 node2 oraasm(asm)[33615]: INFO: Oracle ASM is not running
Dec 19 12:44:45 node2 journal: Oracle Clusterware: 2019-12-19 12:44:45.188#012[(33714)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 33714
Dec 19 12:44:46 node2 oraasm(asm)[33615]: INFO: Oracle ASM is not running
Dec 19 12:44:47 node2 crmd[6925]:  notice: Result of start operation for asm on node2: 0 (ok)
...
Dec 19 12:44:47 node2 crmd[6925]:  notice: Initiating start operation oracle_start_0 locally on node2
Dec 19 12:44:49 node2 oracle(oracle)[34417]: ERROR: oracle HDMDB can not be mounted (status: )
Dec 19 12:44:49 node2 lrmd[6922]:  notice: oracle_start_0:34417:stderr [ ocf-exit-reason:oracle HDMDB can not be mounted (status: ) ]
Dec 19 12:44:49 node2 crmd[6925]:  notice: Result of start operation for oracle on node2: 1 (unknown error)

Resolution

Red Hat Enterprise Linux 7
  • The issue is being tracked with bugzilla 1695024: Bug 1695024 - oraasm: false positive monitoring op result [RHEL 7] (RHEL 7 7.7.0). As of Wed, November 11 2020, the status of bugzilla 1695024 is CLOSED. This bug has been closed because the problem described is an issue that will not be fixed. An explanation of why this resolution is set to WONTFIX should be in the bugzilla and if you cannot access the bug or you want further information contact Red Hat support.

Red Hat Enterprise Linux 8


This issue is being tracked in private Jira [RHEL-137744](https://issues.redhat.com/browse/RHEL-137744). As of 25 December 2025, this bug is in NEW state.

If you would like to track the progress of this bug, please This content is not included.open a case with Red Hat Global Support Services.

Workaround


Add an `ocf:heartbeat:Delay` resource (with the `startdelay` attribute set) between the `oraasm` resource and any resources that depend on it. Note that the `Delay` resource's operation timeouts may need to be increased if the configured delay is close to or greater than 30 seconds. 
# pcs resource describe ocf:heartbeat:Delay
ocf:heartbeat:Delay - Waits for a defined timespan

This script is a test resource for introducing delay.

Resource options:
  startdelay: How long in seconds to delay on start operation.
  stopdelay: How long in seconds to delay on stop operation. Defaults to "startdelay" if unspecified.
  mondelay: How long in seconds to delay on monitor operation. Defaults to "startdelay" if unspecified.

Default operations:
  start: interval=0s timeout=30s
  stop: interval=0s timeout=30s
  monitor: interval=10s timeout=30s

Root Cause

The start operation of the oraasm resource agent consists of two main steps:

Step 1: Run the ASM init script (/etc/init.d/ohasd start).
Step 2: After Step 1 returns, run crsctl check has | grep -q \"CRS-4638\"" once per second until the output of crsctl check has contains the string "CRS-4638".

If crsctl check has prints "CRS-4638: Oracle High Availability Services is online", then Pacemaker considers the resource to be finished starting. However, Oracle may report that High Availability Services is online before ASM has actually finished starting.

oraasm_start() {
    # if resource is already running, no need to continue code after this.
    if oraasm_monitor; then
        ocf_log info "Oracle ASM is already running"
        return $OCF_SUCCESS
    fi

    ocf_run -q /etc/init.d/ohasd start

    while ! oraasm_monitor; do
        sleep 1
    done

    return $OCF_SUCCESS
}
...
oraasm_monitor() {
        su - $OCF_RESKEY_user -c ". $ORA_ENVF; crsctl check has | grep -q \"CRS-4638\""
        case "$?" in
                0)
                        rc=$OCF_SUCCESS
                        ;;
                1)
                        rc=$OCF_NOT_RUNNING
                        ocf_log info "Oracle ASM is not running"
                        ;;
                *)
                        rc=$OCF_ERR_GENERIC
                        ;;
        esac
        return $rc
}

Diagnostic Steps

Find that some Oracle ASM processes are not running immediately after the oraasm start operation completes successfully.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.