Qpid throws JERR__AIO: AIO error. (io_queue_init() failed: errno=11 (Resource temporarily unavailable) error when attempting to create a queue

Environment

• Red Hat Enterprise MRG Messaging
• Red Hat Satellite 6
• common denominator: qpid-cpp-server-store or qpid-cpp-server-linearstore package installed

Issue

• Qpid throws following error when attempting to create a queue

    JERR__AIO: AIO error. (io_queue_init() failed:  errno=11 (Resource temporarily unavailable)
  

Resolution

Increase fs.aio-max-nr from the default of 65536 and restart qpidd service. Use whatever value higher than number of durable queues multiplied by 33 - value 1000000 is used as a reference below. For Satellite6 use case, number of durable queues is slightly higher than number of Content Hosts with katello agent installed.

• append the config change to /etc/sysctl.conf:

    fs.aio-max-nr = 1000000
  

• load/apply the change and restart qpidd:

    # sysctl -p
    # service qpidd restart
  

Root Cause

Qpid issues the error from the following code path:

pmgr::initialize(...)
\-> aio::queue_init(int max_aio_evts, io_context_t *ctxp)
  \-> ::io_queue_init(int maxevents, io_context_t *ctxp)  <- part of libaio
    \-> io_setup()

When we setup this request we are asking for max_aio_evts worth of AIO
What is max_aio_evts?

const u_int16_t max_aio_evts = _cache_num_pages + _jc->num_jfiles();

• _cache_num_pages is the number of pages in one journal size (--file-size in qpid-config option)
• _jc->num_jfiles() is the number of journal files (--file-count in qpid-config option)

Therefore if your configuration has --file-size=8 and --file-count=4 this sets max_aio_evts = 8+4 = 12
If the number of available slots is less than the max_aio_evts request then the request will fail

Diagnostic Steps

• The following errors are found in the qpid logs

    2012-09-21 10:00:19 error Unexpected exception: Queue queue_name: create() failed: jexception 0x0103 pmgr::initialize() threw JERR__AIO: AIO error. (io_queue_init() failed:  errno=11 (Resource temporarily unavailable)) (MessageStoreImpl.cpp:538)
    2012-09-21 10:00:19 error Connection 10.1.2.3:21313-10.1.2.5:34877 closed by error: Queue queue_name: create() failed: jexception 0x0103 pmgr::initialize() threw JERR__AIO: AIO error. (io_queue_init() failed:  errno=11 (Resource temporarily unavailable)) (MessageStoreImpl.cpp:538)(501)
  

Examine the values of

$ cat /proc/sys/fs/aio-nr 
65528

$ cat /proc/sys/fs/aio-max-nr 
65536

• This leaves us with 65536 - 65528 = 8 available AIO event slots
• Calculate the value of max_aio_evts. --file-size=8 and --file-count=4 this sets max_aio_evts = 8+4 = 12
• The number of available slots (8) is less than 12 requested so the request fails
• If you sum the number of journal files and the number of pages across all your queues etc, then you will arrive at the absolute minimum number you need to set fs.aio-max-nr to

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.