JBoss SOA-P very slow to start after adjusting JMS destination paging parameters

Environment

• JBoss Enterprise SOA Platform (SOA-P)

Issue

• We adjusted the paging parameters (i.e. FullSize, PageSize, DownCacheSize) for some of our JMS destinations to avoid OutOfMemoryErrors when starting the server, but now the server is starting very slowly.After a pause of 5 minutes some errors are thrown and the server boots but our application is crippled because a JMS destination didn't deploy.

Resolution

Increase the default transaction timeout (How do I modify the default transaction timeout in JBoss EAP?) to a value large enough so that the errors are not thrown and the destination deploys.  After the destination deploys properly reduce the transaction timeout to its default value.

Root Cause

During normal operation of the JMS destination cache the PAGE_ORD column in the JBM_MSG_REF table is updated appropriately.  Only messages which have been "paged" to the database have a non-null PAGE_ORD value which means there will always be <FullSize> messages which have a null PAGE_ORD.  However, when the FullSize is decreased, JBoss Messaging must then update all the messages beyond the current FullSize with a valid PAGE_ORD.  This can take a long time depending on how many messages need to be updated.  All these updates happen in a transaction so when the transaction times out errors are logged and the destination fails to deploy.

Diagnostic Steps

• Get a thread dump during the pause and analyze it:
  • How do I create a Java heap dump?
  • How do I analyze a Java heap dump?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.