JBoss ON storage nodes fail to form the cluster due to rpc_address set to the hostname instead to the IP address as configured in the rhq-server.properties file

Environment

• Red Hat JBoss Operations Network (ON) 3.3
• The host name definition resolves to the local machine's loopback interface - 127.0.0.1

Issue

• JBoss ON storage node is not able to form the cluster with another storage nodes due to rpc_address being set to the host name instead of the IP address that is configured in the rhq-server.properties file;

Resolution

If the affected storage node has already been installed and has been participating in the cluster, you will need to manually modify the storage node's internal configuration directly. If you have not yet installed the storage node or are able to delete the existing rhq-storage installation and re-install, you can apply the latest JBoss Operations Network 3.3 update and define the rhq.storage.rpc-address and rhq.storage.listen-address properties in rhq-storage.properties.

• For fresh install of JBoss Operations Network Storage Node:

  1. Update to JBoss Operations Network 3.3 Update 11 or later.

  2. Modify <RHQ_SERVER_HOME>/bin/rhq-storage.properties and set rhq.storage.rpc-address and rhq.storage.listen-address to appropriate values.

     The addresses must be fully resolvable and reachable from all JBoss Operations Network servers and storage nodes in the cluster.

  3. Perform the JBoss Operations Network Storage Node installation as normal.

• For existing JBoss Operations Network Storage Node:

  1. Make a backup or the <RHQ_SERVER_HOME>/rhq-storage directory.

  2. Modify <RHQ_SERVER_HOME>/rhq-stroage/conf/cassandra.yaml and change the value of rpc_address.

  3. Update to JBoss Operations Network 3.3 Update 11 or later.

  4. Modify <RHQ_SERVER_HOME>/bin/rhq-storage.properties and set rhq.storage.rpc-address and rhq.storage.listen-address to appropriate values.

     Although the value will not be used, this step is still recommended in case a future re-install or restoration is performed.

Root Cause

The JBoss Operations Network Storage Node uses the default or defined value of rhq.storage.hostname for its RPC network bind address and its listen address. If the default host name is used and it is not resolvable or cannot be routed, the storage node will fail to start or will not be available to other storage nodes or servers in a high-availability cluster.

This issue has been reported in the This content is not included.Red Hat Bugzilla 1517717.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.