How to manually abort a pending transaction of a malfuncting client in qpid broker?

Solution Verified - Updated

Environment

  • MRG Messaging (any version)

Issue

  • exception "JERR_MAP_LOCKED: Record ID locked by a pending transaction" raised when attempting to consume messages from a queue
  • per the log, some qpid client apparently forgot to commit/rollback its transaction but we don't know which one

How to identify what session or client is responsible for it? An how to manually abort/rollback the pending transaction?

Resolution

Apparently some consumer on some session had to started a transaction but has not finished it (neither committed or aborted it by itself). The less offensive possibility to abort the transaction is to drop the underlying connection for the session.

To do so, let first identify the connection and client, and then drop the connection from qpid-tool.

To identify the malfunctioning client

To identify the malfunctioning client and its connection, let run:

qpid-stat -u | grep MyQueue

One should see an output like:

  MyQueue  MyQueue  qpid.127.0.0.1:5672-127.0.0.1:42648  qpid-receive  5789            Y            WINDOW      10         10

where:

  • the first column is the name of subscription (may differ from the queue name)
  • second "MyQueue" is the queue name
  • third is the TCP connection (the "qpid" might not be there, it depends on qpidd version)
  • fourth and fivth is the remote process name and its PID (it might be blank)
  • the latest figure is the number of unacknowledged messages.

If there are more consumers in qpid-stat output, check after a while which one is stalled in message consumption. If in doubts, preventively drop all potentially misbehaving clients.

To drop the connection

Note that by dropping the connection from the malfunctioning client, all the unacknowledged messages will be released and re-delivered to another consumer of that queue, with "redelivered" flag set on (to mark these messages have already been received by some consumer that hasn't acknowledged or rejected these).

To drop the connection identified as above, use qpid-tool as follows:

  • run qpid-tool
  • wait 10 seconds to get the tool populated by broker's data
  • type "list connection":
qpid: list connection
Object Summary:
    ID   Created   Destroyed  Index
    ===================================================================
    118  08:56:45  -          145.qpid.127.0.0.1:5672-127.0.0.1:42648
    119  09:00:04  -          145.qpid.127.0.0.1:5672-127.0.0.1:42651
qpid:
  • find the relevant TCP connection there and its ID (in our case it is 118)
  • now call QMF method to drop the connection:
qpid: call 118 close
qpid: OK (0) - {}

Now the disconnected client will get an exception like "closed connection: 320, Closed by Management Request" and it is disconnected from the broker.

Diagnostic Steps

Exception

Queue MyQueue: async_dequeue() failed: jexception 0x0b02 wmgr::dequeue_check() threw JERR_MAP_LOCKED: Record ID locked by a pending transaction. (drid=0xdf261) (MessageStoreImpl.cpp:1419)

raised when attempting to consume messages from a queue.

To mimic the behaviour, let run:

  • in first terminal:
qpid-receive -a "MyQueue; {create:always, node:{durable:true}}" -m 100 --tx 50 -f
  • in second terminal:
qpid-send -a MyQueue -m 10 --durable=yes

The running qpid-receive has now acquired messages in a transaction but hasn't committed that activity yet.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.