How to capture crash dump (vmcore) upon Oracle RAC 11g R1 (or earlier) node eviction ?

Environment

• Red Hat Enterprise Linux 4
• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6
• Oracle RAC 11g R1 or earlier (11.1.0.7 or earlier)
• Oracle Cluster Services Synchronization Daemon ( CSSD )

Issue

• Why during Oracle RAC node eviction Red Hat Enterprise Linux server always reboot without capturing vmcore ?
• How to configure Oracle CSSD service to capture crash dump (vmcore) during Oracle RAC 11g R1 (or earlier) node eviction ?

Resolution

• By default Oracle Cluster Services Synchronization Daemon (CSSD) reboot the server during Oracle RAC node eviction.

# grep -i FAST_REBOOT /etc/init.d/init.cssd
FAST_REBOOT="$ECHO b > /proc/sysrq-trigger"

• To capture vmcore during Oracle RAC node eviction, setup netdump/diskdump (RHEL-4) or kdump (RHEL-5/6) service on Red Hat Enterprise Linux.
• Edit and change the "FAST_REBOOT" line in Oracle Cluster Services Synchronization Daemon (CSSD) /etc/init.d/init.cssd file.
  Eg:

# vim /etc/rc.d/init.d/init.cssd
FAST_REBOOT="$ECHO c > /proc/sysrq-trigger"

• Restart "cssd" service for the changes to effect.
  Eg:

#/etc/init.d/init.cssd start

Note: As Oracle Cluster Services Synchronization Daemon (CSSD) is a part Oracle RAC it is advisable to contact Oracle Technical Support team before making above changes.

• An alternative would be to create a Systemtap module that calls a panic() whenever a emergency_shutdown() function call is made on Oracle RAC.

Note: See the How to capture kernel crash dump (vmcore) upon Oracle RAC 11gR2 or Oracle RAC 12 node eviction ? article for configuring the behaviour on node eviction for Oracle RAC 11gR2 and later.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.