Manifest refresh/import in Satellite6 fails with "detached entity passed to persist: org.candlepin.model.EntitlementCertificate" error

Solution Verified - Updated 14 Jun 2024

Environment

Red Hat Satellite 6.3 (other versions might be affected as well)

Issue

an attempt to refresh or import manifest fails with javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate error
the failure happened a few minutes after a round hour (like 09:03 or 12:05 but not 16:46)

Resolution

For a permanent solution, wait until This content is not included.the underlying bug is fixed.

For a reactive workaround, run new manifest refresh / import. There is just very small chance to hit the race condition.

For a preventive workaround, disable ExpiredPoolsJob execution during the manifest refresh / import. Notice that this require two restarts of candlepin, though:

add to /etc/candlepin/candlepin.conf:

pinsetter.org.candlepin.pinsetter.tasks.ExpiredPoolsJob.schedule=0 0 0 1 1 ?

(to set up triggering the task on New Years Eve midnight only)

restart candlepin:

systemctl restart tomcat.service

import or refresh manifest now
revert back the change in config and restart tomcat service again (the job is worth to be regularly running)

FYI default frequency of the job execution is "0 0 0/1 * * ?", i.e. every hour.

For more KB articles/solutions related to Red Hat Satellite 6.x Manifest Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Manifest Issues

Root Cause

ExpiredPoolsJob was regularly invoked at the same time when manifest refresh or import identified expired pools and was about to delete them as well. Those two actions were trying to manipulate with same objects concurrently, what failed.

Diagnostic Steps

/var/log/candlepin/candlepin.log has failed manifest import request like:

2019-05-06 08:59:50,474 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=, csid=92b17d79] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/ORG/imports
..
2019-05-06 09:03:42,688 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=ORG, csid=92b17d79] INFO  org.candlepin.controller.CandlepinPoolManager - Recomputing status for 475 consumers
2019-05-06 09:03:52,319 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=ORG, csid=92b17d79] ERROR org.candlepin.sync.Importer - Failed to import archive
javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1303) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:564) ~[AbstractHibernateCurator.class:na]
        at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:2245) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:1998) ~[CandlepinPoolManager.class:na]
        at org.candlepin.controller.CandlepinPoolManager.refreshPoolsWithRegeneration(CandlepinPoolManager.java:359) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.Refresher.run(Refresher.java:159) ~[Refresher.class:na]
        at org.candlepin.sync.Importer.importObjects(Importer.java:595) [Importer.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
        at org.candlepin.sync.Importer.doExport(Importer.java:415) [Importer.class:na]
        at org.candlepin.sync.Importer.loadExport(Importer.java:176) [Importer.class:na]
..

While at very similar time, the log also contains:

2019-05-06 09:03:57,076 [thread=QuartzScheduler_Worker-15] [job=ExpiredPoolsJob-76d0e223-d4ee-421e-ad17-f07925073177, org=, csid=] INFO  org.candlepin.controller.CandlepinPoolManager - Recomputing status for 475 consumers
2019-05-06 09:04:05,497 [thread=QuartzScheduler_Worker-15] [job=ExpiredPoolsJob-76d0e223-d4ee-421e-ad17-f07925073177, org=, csid=] ERROR org.candlepin.pinsetter.tasks.KingpinJob - Persistence exception caught running pinsetter task. Attempt: 0, Refire: true
javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1303) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:564) ~[AbstractHibernateCurator.class:na]
        at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:2245) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:1998) ~[CandlepinPoolManager.class:na]
        at org.candlepin.controller.CandlepinPoolManager.cleanupExpiredPoolsImpl(CandlepinPoolManager.java:508) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.CandlepinPoolManager.cleanupExpiredPools(CandlepinPoolManager.java:480) ~[CandlepinPoolManager.class:na]
        at org.candlepin.pinsetter.tasks.ExpiredPoolsJob.toExecute(ExpiredPoolsJob.java:49) ~[ExpiredPoolsJob.class:na]

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

candlepin

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.