A storage node in operation mode "MAINTENANCE" cannot be undeployed

Environment

• Red Hat JBoss Operations Netwok (ON) 3.3
• Storage node with:
  • Cluster Status: NORMAL
  • Availability: UP
  • Operation Mode: MAINTENANCE

Issue

• A storage node in operation mode "MAINTENANCE" cannot be undeployed;
• An attempt to undeploy a storage node that has operation mode "MAINTENANCE" throws the following error in the server.log:

Caused by: java.lang.RuntimeException: Cannot undeploy StorageNode[id=10001, address=myStorage.Node.com, cqlPort=9142, operationMode=MAINTENANCE, mtime=1433524868590]

Resolution

This is a bug and it has been reported in the This content is not included.Red Hat Bugzilla 1272328.

To workaround the issue, the following database query can be executed. Please note that a complete database backup should be performed in case of error as this workaround is not supported by Red Hat Support.

UPDATE rhq_storage_node 
SET operation_mode = 'NORMAL',
    resource_op_hist_id = NULL, 
    error_msg = NULL 
WHERE id IN ( SELECT id FROM rhq_storage_node WHERE operation_mode = 'MAINTENANCE' AND resource_id IS NOT NULL);

The query will change the Operation Mode of the storage node from MAINTENANCE to NORMAL. An attempt to undeploy the storage node should work.

Root Cause

Undeployment of the storage node with Operation Mode MAINTENANCE will fail with "Cannot undeploy ..." message as per following code:

308     public void undeployStorageNode(Subject subject, StorageNode storageNode) {
309         StorageNodeCriteria c = new StorageNodeCriteria();
310         c.addFilterId(storageNode.getId());
311         c.fetchResource(true);
312         List<StorageNode> storageNodes = storageNodeManager.findStorageNodesByCriteria(subject, c);
313         if (storageNodes.isEmpty()) {
314             throw new RuntimeException("Storage node not found, can not undeploy " + storageNode);
315         }
316         storageNode = storageNodes.get(0);
317 
318         switch (storageNode.getOperationMode()) {
319         case INSTALLED:
320             storageNodeManager.resetInNewTransaction();
321             storageNodeOperationsHandler.uninstall(subject, storageNode);
322             break;
323         case ANNOUNCE:
324         case BOOTSTRAP:
325             storageNodeManager.resetInNewTransaction();
326             storageNodeOperationsHandler.unannounceStorageNode(subject, storageNode);
327             break;
328         case ADD_MAINTENANCE:
329         case NORMAL:
330         case DECOMMISSION:
331             storageNodeManager.resetInNewTransaction();
332             storageNodeOperationsHandler.decommissionStorageNode(subject, storageNode);
333             break;
334         case REMOVE_MAINTENANCE:
335             storageNodeManager.resetInNewTransaction();
336             storageNodeOperationsHandler.performRemoveNodeMaintenance(subject, storageNode);
337             break;
338         case UNANNOUNCE:
339             storageNodeManager.resetInNewTransaction();
340             storageNodeOperationsHandler.unannounceStorageNode(subject, storageNode);
341             break;
342         case UNINSTALL:
343             storageNodeManager.resetInNewTransaction();
344             storageNodeOperationsHandler.uninstall(subject, storageNode);
345             break;
346         default:
347             // TODO what do we do with/about maintenance mode
348             throw new RuntimeException("Cannot undeploy " + storageNode);
349         }
350     }

Diagnostic Steps

• Execute the following query to confirm that storage node is in maintenance mode:

    SELECT * FROM rhq_storage_node;
  

  Confirm that similar to the following is returned:

    10002   storageNode1     9142    NORMAL          1426613258016   1426613258016           0       10258           4.12.0.JON330GA
    10001   storageNode2     9142    MAINTENANCE     1426612947921   1426612947921           1       10133           4.12.0.JON330GA
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.