'java.lang.OutOfMemoryError: Metaspace' related to ByteBuddy / HibernateProxy classes in JBoss EAP
Environment
- Red Hat JBoss Enterprise Application Platform (EAP) 7
- Hibernate 5
Issue
-
Application is repeatedly (during its lifecycle) calling
javax.persistence.Persistence.createEntityManagerFactory -
After some time, encountering
java.lang.OutOfMemoryError: Metaspace... javax.persistence.PersistenceException: [PersistenceUnit: my-persistence-unit] Unable to build Hibernate SessionFactory at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.persistenceException(EntityManagerFactoryBuilderImpl.java:1327) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1253) at org.hibernate.jpa.HibernatePersistenceProvider.createEntityManagerFactory(HibernatePersistenceProvider.java:56) at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:80) at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:55) ... Caused by: org.hibernate.MappingException: Could not get constructor for org.hibernate.persister.entity.SingleTableEntityPersister at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:123) at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:77) at org.hibernate.metamodel.internal.MetamodelImpl.initialize(MetamodelImpl.java:154) at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:295) at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:467) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1250) ... Caused by: java.lang.OutOfMemoryError: Metaspace at sun.misc.Unsafe.defineClass(Native Method) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$Enabled.defineClass(ClassInjector.java:1987) at net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe.injectRaw(ClassInjector.java:1774) at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject(ClassInjector.java:112) at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$ForUnsafeInjection.load(ClassLoadingStrategy.java:546) at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:100) at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:5662) at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.lambda$load$0(ByteBuddyState.java:187) at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState$$Lambda$1163/2000834768.call(Unknown Source) at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:152) at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:365) at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:174) at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:376) at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.load(ByteBuddyState.java:183) at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.loadProxy(ByteBuddyState.java:107) at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyProxyHelper.buildProxy(ByteBuddyProxyHelper.java:54) at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyProxyFactory.postInstantiate(ByteBuddyProxyFactory.java:62) at org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFactory(PojoEntityTuplizer.java:103) at org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:155) at org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:51) ... at org.hibernate.tuple.entity.EntityTuplizerFactory.constructTuplizer(EntityTuplizerFactory.java:88) at org.hibernate.tuple.entity.EntityTuplizerFactory.constructDefaultTuplizer(EntityTuplizerFactory.java:116) at org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:413) at org.hibernate.persister.entity.AbstractEntityPersister.<init>(AbstractEntityPersister.java:557) at org.hibernate.persister.entity.SingleTableEntityPersister.<init>(SingleTableEntityPersister.java:124) -
Heap dump shows thousands of unique classes like
org.myapp.somepackage.MyEntity$HibernateProxy$0GW5bPN9(where the alphanumeric suffix is distinct)
Resolution
-
Session factories are generally expected to serve as singletons. They are not lightweight objects. Creation is often expensive (i.e. takes some time) due to the calculation of metadata for the object model and each instance may be associated with memory use (proportional to the size and structure of the model) for the metadata.
-
In most EJB scenarios, it is usually recommended to directly inject a container-managed entity manager and avoid the need to access the entity manager factory.
@PersistenceContext(name = "my-persistence-unit") EntityManager myEntityManager; -
Reuse the container managed entity manager factory using one of the approaches below.
-
Inject the container-managed entity manager factory (this is the simplest/recommended approach if directly using the factory).
@PersistenceUnit(unitName = "my-persistence-unit") EntityManagerFactory myEntityManagerFactory; -
Lookup the container-managed entity manager factory.
-
Add to the
persistence.xmla property with a unique JNDI url for each unit for which the factory is required.<property name="jboss.entity.manager.factory.jndi.name" value="java:/my-persistence-unit"/> -
Use a call to
EntityManagerFactory.class.cast(new InitialContext().lookup("java:/my-persistence-unit"))to retrieve the entity manager factory.- This instance may be cached and re-used for the lifetime of the container-managed persistence unit.
-
-
Note that you must not close an injected or JNDI retrieved factory (because it is owned by the container) but you must explicitly close any entity manager instance created explicitly using the factory reference.
-
-
This issue will be addressed in a future release1.
Root Cause
- During each creation of an entity manager factory / session factory, proxies must be created for use by bytecode assistance mechanisms
- The proxy classes cannot be shared across factory instances
- Once created these classes cannot be unloaded
- This is a known defect (Content from hibernate.atlassian.net is not included.HHH-14694).
Diagnostic Steps
Confirming the issue using Eclipse Memory Analysis Tool (MAT)
- With a heap dump loaded, open the Histogram view
- In the class name filter, enter "HibernateProxy" and click
Enter - The usual symptom is that thousands of unique classes like
org.myapp.somepackage.MyEntity$HibernateProxy$0GW5bPN9(perhaps for different entity types but where many examples for a single entity class type are found with unique alphanumeric suffixes)
Confirming excessive SessionFactory creation
Enable DEBUG for org.hibernate.internal.SessionFactoryImpl and check for messages like the below
... DEBUG [org.hibernate.internal.SessionFactoryImpl] ... Building session factory
...
... DEBUG [org.hibernate.internal.SessionFactoryImpl] ... Instantiated session factory
Identifying the origin of excessive SessionFactory creation
Deploy Byteman with a rule like the below to obtain stack traces for SessionFactory instance creation
RULE org.hibernate.internal.SessionFactoryImpl
CLASS org.hibernate.internal.SessionFactoryImpl
METHOD <init>
IF true
DO traceStack("[BMAN] ", 15);
ENDRULE
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.