qpid broker fails to start on store recovery: with error "Cannot allocate memory"

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 6 or 7
  • Red Hat Enterprise MRG Messaging
    • store module used - both legacy store (qpid-cpp-server-store) and linear store (qpid-cpp-server-linearstore) could be affected

Issue

  • after stopping qpid broker, unable to start it again
  • several particular error messages seen, most often with Cannot allocate memory:
    • Error on recovery (MessageStoreImpl.cpp:701): Dbc::get: Cannot allocate memory
    • JERR__AIO: AIO error. (io_queue_init() failed: errno=12 (Cannot allocate memory)
    • JERR_JDIR_STAT: File stat failed. (file="/var/lib/qpidd/.qpidd/qls/jrnl/pulp.agent.ea302561-219c-42a7-bd18-2e3038349769" errno=2 (No such file or directory)
    • JERR__MALLOC: Buffer memory allocation failed. (posix_memalign(): alignment=4096 size=1048576 errno=12 (Cannot allocate memory))
  • the machine has much memory that is not used
  • ulimit or kernel tunables tuned appropriatelly
  • after wiping out provisioning (under /var/lib/qpidd/ directory), broker can start (but without the stored provisioning and journals - unacceptable solution)

Resolution

Tune Berkeley DB to allow more objects and locks, by setting some variables in DB_CONFIG file in the specified directory.

Create file /var/lib/qpidd/.qpidd/qls/dat2/DB_CONFIG owned/readable by qpidd user with content:

# cat /var/lib/qpidd/.qpidd/qls/dat2/DB_CONFIG
set_lk_max_locks 10000
set_lk_max_objects 10000
#

Restart the broker now.

Learn More


See the [Red Hat Satellite Installation guide](https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/Installation_Guide/sect-Red_Hat_Satellite-Installation_Guide-Prerequisites.html#sect-Red_Hat_Satellite-Installation_Guide-Prerequisites-Large_deployments) for more on considerations for large deployments.

Root Cause

The cryptic message "Dbc::get: Cannot allocate memory" comes from Berkeley database (BDB) hitting default limit on number of 1000 concurrently allowed locks and/or objects. Probably due to the ordering of write events to the Berkeley DB underneath, the store recovery required more locks.

Increasing the default values via DB_CONFIG causes no harm while resolves the problem.

Diagnostic Steps

Qpid logs:

2015-02-12 14:14:18 [Broker] critical Unexpected error: Error on recovery (MessageStoreImpl.cpp:701): Dbc::get: Cannot allocate memory
2015-02-12 14:14:18 [Broker] critical Unexpected error: Daemon startup failed: Error on recovery (MessageStoreImpl.cpp:701): Dbc::get: Cannot allocate memory

When starting broker with traces/debugs enabled, no error might be hit and the broker can start properly.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.