'java.lang.OutOfMemoryError: Metaspace' related to ByteBuddy / HibernateProxy classes in JBoss EAP

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 7
  • Hibernate 5

Issue

  • Application is repeatedly (during its lifecycle) calling javax.persistence.Persistence.createEntityManagerFactory

  • After some time, encountering java.lang.OutOfMemoryError: Metaspace

    ... javax.persistence.PersistenceException: [PersistenceUnit: my-persistence-unit] Unable to build Hibernate SessionFactory
            at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.persistenceException(EntityManagerFactoryBuilderImpl.java:1327)
            at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1253)
            at org.hibernate.jpa.HibernatePersistenceProvider.createEntityManagerFactory(HibernatePersistenceProvider.java:56)
            at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
            at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
    ...
    Caused by: org.hibernate.MappingException: Could not get constructor for org.hibernate.persister.entity.SingleTableEntityPersister
            at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:123)
            at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:77)
            at org.hibernate.metamodel.internal.MetamodelImpl.initialize(MetamodelImpl.java:154)
            at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:295)
            at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:467)
            at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1250)
    ...
    Caused by: java.lang.OutOfMemoryError: Metaspace
            at sun.misc.Unsafe.defineClass(Native Method)
            at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$Enabled.defineClass(ClassInjector.java:1987)
            at net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe.injectRaw(ClassInjector.java:1774)
            at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject(ClassInjector.java:112)
            at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$ForUnsafeInjection.load(ClassLoadingStrategy.java:546)
            at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:100)
            at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:5662)
            at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.lambda$load$0(ByteBuddyState.java:187)
            at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState$$Lambda$1163/2000834768.call(Unknown Source)
            at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:152)
            at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:365)
            at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:174)
            at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:376)
            at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.load(ByteBuddyState.java:183)
            at org.hibernate.bytecode.internal.bytebuddy.ByteBuddyState.loadProxy(ByteBuddyState.java:107)
            at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyProxyHelper.buildProxy(ByteBuddyProxyHelper.java:54)
            at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyProxyFactory.postInstantiate(ByteBuddyProxyFactory.java:62)
            at org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFactory(PojoEntityTuplizer.java:103)
            at org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:155)
            at org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:51)
            ...
            at org.hibernate.tuple.entity.EntityTuplizerFactory.constructTuplizer(EntityTuplizerFactory.java:88)
            at org.hibernate.tuple.entity.EntityTuplizerFactory.constructDefaultTuplizer(EntityTuplizerFactory.java:116)
            at org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:413)
            at org.hibernate.persister.entity.AbstractEntityPersister.<init>(AbstractEntityPersister.java:557)
            at org.hibernate.persister.entity.SingleTableEntityPersister.<init>(SingleTableEntityPersister.java:124)
    
    
  • Heap dump shows thousands of unique classes like org.myapp.somepackage.MyEntity$HibernateProxy$0GW5bPN9 (where the alphanumeric suffix is distinct)

Resolution

  • Session factories are generally expected to serve as singletons. They are not lightweight objects. Creation is often expensive (i.e. takes some time) due to the calculation of metadata for the object model and each instance may be associated with memory use (proportional to the size and structure of the model) for the metadata.

  • In most EJB scenarios, it is usually recommended to directly inject a container-managed entity manager and avoid the need to access the entity manager factory.

    @PersistenceContext(name = "my-persistence-unit")
    EntityManager myEntityManager;
    
  • Reuse the container managed entity manager factory using one of the approaches below.

    • Inject the container-managed entity manager factory (this is the simplest/recommended approach if directly using the factory).

      @PersistenceUnit(unitName = "my-persistence-unit")
      EntityManagerFactory myEntityManagerFactory;
      
    • Lookup the container-managed entity manager factory.

      • Add to the persistence.xml a property with a unique JNDI url for each unit for which the factory is required.

                                <property name="jboss.entity.manager.factory.jndi.name" value="java:/my-persistence-unit"/>
        
      • Use a call to EntityManagerFactory.class.cast(new InitialContext().lookup("java:/my-persistence-unit")) to retrieve the entity manager factory.

        • This instance may be cached and re-used for the lifetime of the container-managed persistence unit.
    • Note that you must not close an injected or JNDI retrieved factory (because it is owned by the container) but you must explicitly close any entity manager instance created explicitly using the factory reference.

  • This issue will be addressed in a future release1.

Root Cause

  • During each creation of an entity manager factory / session factory, proxies must be created for use by bytecode assistance mechanisms
  • The proxy classes cannot be shared across factory instances
  • Once created these classes cannot be unloaded
  • This is a known defect (Content from hibernate.atlassian.net is not included.HHH-14694).

Diagnostic Steps

Confirming the issue using Eclipse Memory Analysis Tool (MAT)

  • With a heap dump loaded, open the Histogram view
  • In the class name filter, enter "HibernateProxy" and click Enter
  • The usual symptom is that thousands of unique classes like org.myapp.somepackage.MyEntity$HibernateProxy$0GW5bPN9 (perhaps for different entity types but where many examples for a single entity class type are found with unique alphanumeric suffixes)

Confirming excessive SessionFactory creation

Enable DEBUG for org.hibernate.internal.SessionFactoryImpl and check for messages like the below

... DEBUG [org.hibernate.internal.SessionFactoryImpl] ... Building session factory
...
... DEBUG [org.hibernate.internal.SessionFactoryImpl] ... Instantiated session factory

Identifying the origin of excessive SessionFactory creation

Deploy Byteman with a rule like the below to obtain stack traces for SessionFactory instance creation

RULE org.hibernate.internal.SessionFactoryImpl
CLASS org.hibernate.internal.SessionFactoryImpl
METHOD <init>
IF true
DO traceStack("[BMAN] ", 15);
ENDRULE
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.