JBoss ON Storage Node not installed with `rhqctl upgrade`

Environment

• Red Hat JBoss Operations Network (ON) 3.3
• upgrading from ON version 3.1.2

Issue

• Running rhqctl.bat upgrade –-from-server-dir d:\JBoss\jon-server-3.1.2.GA -–from-agent-dir d:\JBoss\rhq-agent and receive error on storage node:

10:42:50,000 ERROR [org.rhq.server.control.RHQControl] The storage node upgrade failed with exit code [2]: java.lang.RuntimeException: The storage node upgrade failed with exit code [2]
	at org.rhq.server.control.command.Upgrade.upgradeStorage(Upgrade.java:290) [rhq-server-control-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at org.rhq.server.control.command.Upgrade.exec(Upgrade.java:202) [rhq-server-control-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at org.rhq.server.control.ControlCommand.exec(ControlCommand.java:181) [rhq-server-control-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at org.rhq.server.control.RHQControl.exec(RHQControl.java:106) [rhq-server-control-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at org.rhq.server.control.RHQControl.main(RHQControl.java:357) [rhq-server-control-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
	at org.jboss.modules.Module.run(Module.java:270) [jboss-modules.jar:1.2.0.Final-redhat-1]
	at org.jboss.modules.Main.main(Main.java:411) [jboss-modules.jar:1.2.0.Final-redhat-1]

Resolution

This issue has been resolved in JBoss ON 3.3 Update-05.

Root Cause

Upgrade from JON 3.2.0 to JON 3.3.3 failed with "Could not verify that the node is up and running". This is caused by a slow environment:

22:53:18,695 INFO  [org.rhq.storage.installer.StorageInstaller] Starting RHQ Storage Node
22:53:28,836 WARN  [org.rhq.storage.installer.StorageInstaller] Could not verify that the node is up and running.
22:53:28,836 WARN  [org.rhq.storage.installer.StorageInstaller] Check the log file at ../../logs/rhq-storage.log for errors.

There is 10 seconds between "Starting RHQ Storage" and "Could not verify that the node" so if the Storage Node is not started in that time this issue will occur.

This has been logged as This content is not included.Red Hat Bug 1231199.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.