Red Hat Satellite's web UI is not displaying all hypervisors identified by Virt-who

Solution Verified - Updated 13 Jun 2024

Environment

Red Hat Satellite 6.9.4-1

Issue

We have one ESXi host not showing in the Satellite web UI, even though we see no issue from Virth-who logs.
One ESXi host is correctly identified by Virt-who but not shown in Satellite web UI.
In Satellite web UI, all ESXi hosts have an outdated last checkin time.

Resolution

Edit /etc/tomcat/tomcat.conf.
Comment the line:

JAVA_OPTS="-Xms1024m -Xmx4096m"

Add the line:

JAVA_OPTS="-Xms1024m -Xmx6G"

Restart the Tomcat service to apply the change:

# service tomcat restart

Run the following command:

# virt-who --one-shot

Check in Satellite's web UI if it was updated with the missing ESXi and/or the last checkin time of the ESXi hosts was updated.
Check if new Java heap space-related errors are still showing up in /var/log/candlepin/candlepin.log

# grep 'Java heap space' /var/log/candlepin/candlepin.log

If the checks in steps (6) and (7) indicate that the issue still persists, edit /etc/tomcat/tomcat.conf again, and change the line:

JAVA_OPTS="-Xms1024m -Xmx6G"

to:

JAVA_OPTS="-Xms1024m -Xmx8G"

Restart the Tomcat service to apply the change:

# service tomcat restart

Repeat steps (6) and (7) to confirm if the issue was resolved. If the issue still persists, open a case with Red Hat support.

For more KB articles/solutions related to Virt-who and Virtual Datacenter (VDC) Subscriptions Issues, please refer to the Consolidated Troubleshooting Article for Virt-who and Virtual Datacenter (VDC) Subscriptions Issues

Diagnostic Steps

The following warning is recurring in /var/log/candlepin/candlepin.log on the Satellite server:

2022-08-08 09:08:34,109 [thread=Thread-4 (ActiveMQ-client-global-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - Java heap space
java.lang.OutOfMemoryError: Java heap space
	at com.google.common.io.ByteStreams.toByteArrayInternal(ByteStreams.java:176)
	at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:221)
	at org.candlepin.util.JSSX509CRLStreamWriter.preScan(JSSX509CRLStreamWriter.java:220)
	at org.candlepin.util.CrlFileUtil.updateCRLFile(CrlFileUtil.java:240)
	at org.candlepin.util.CrlFileUtil.batchSyncCRLWithDB(CrlFileUtil.java:377)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:70)
	at org.candlepin.util.CrlFileUtil.syncCRLWithDB(CrlFileUtil.java:428)
	at org.candlepin.async.tasks.CRLUpdateJob.execute(CRLUpdateJob.java:87)
	at org.candlepin.async.JobManager.executeJob(JobManager.java:1322)
	at org.candlepin.async.JobMessageReceiver$MessageListener.handleMessage(JobMessageReceiver.java:394)
	at org.candlepin.messaging.impl.artemis.ArtemisConsumer$ArtemisMessageForwarder.onMessage(ArtemisConsumer.java:59)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$354/658598758.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

The following warning might also appear in /var/log/candlepin/candlepin.log on the Satellite server:

2022-08-08 09:09:59,507 [thread=Thread-2 (ActiveMQ-client-global-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.utils.actors.OrderedExecutor - input is too large to fit in a byte array
java.lang.OutOfMemoryError: input is too large to fit in a byte array
	at com.google.common.io.ByteStreams.toByteArrayInternal(ByteStreams.java:195)
	at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:221)
	at org.candlepin.util.JSSX509CRLStreamWriter.preScan(JSSX509CRLStreamWriter.java:220)
	at org.candlepin.util.CrlFileUtil.updateCRLFile(CrlFileUtil.java:240)
	at org.candlepin.util.CrlFileUtil.batchSyncCRLWithDB(CrlFileUtil.java:377)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:70)
	at org.candlepin.util.CrlFileUtil.syncCRLWithDB(CrlFileUtil.java:428)
	at org.candlepin.async.tasks.CRLUpdateJob.execute(CRLUpdateJob.java:87)
	at org.candlepin.async.JobManager.executeJob(JobManager.java:1322)
	at org.candlepin.async.JobMessageReceiver$MessageListener.handleMessage(JobMessageReceiver.java:394)
	at org.candlepin.messaging.impl.artemis.ArtemisConsumer$ArtemisMessageForwarder.onMessage(ArtemisConsumer.java:59)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

The following parameter is set in /etc/tomcat/tomcat.conf on the Satellite server:

JAVA_OPTS="-Xms1024m -Xmx4096m"

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.