Use of symbolic links leads to large disk usage in JBoss EAP

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 5.x.x

Issue

• The server configuration directories ($JBOSS_HOME/server/$PROFILE) are linked via symbolic links (symlinks) to another file system. That is, $PROFILE is a symlink.
• An application ear is deployed directly to the $JBOSS_HOME/server/$PROFILE/deploy directory without using symbolic links.
• When a new web browser session is started for the application, the $JBOSS_HOME/server/$PROFILE/tmp/vfs-nested.tmp folder doubles in size causing the file system to fill up very quickly when there are many concurrent user sessions.
• When we remove the symbolic links, everything works as expected and the $JBOSS_HOME/server/$PROFILE/tmp/vfs-nested.tmp folder does not double in size when a new session is created.

• Having problems with the ./tmp path, the subpath vfs-nested.tmp is consuming lots of space. The only fix we're aplying is restart the jboss.
• Vfs-nested.tmp is growing without limits.

Resolution

The are three potential workarounds:

• Avoid the use of symlinks in your JBoss directory hierarchy.
• Deploy an ear as an exploded directory.
• Set the following system properties to the actual server config dir (instead of the symlink):
  • jboss.server.home.dir
  • jboss.server.home.url

For example, if the symlink points to /jboss-configs/default, you would have to add the following in $JBOSS_HOME/bin/run.conf or to the command line:

-Djboss.server.home.dir=/jboss-configs/default -Djboss.server.home.url=file:///jboss-configs/default 

Additionally, one can apply the This content is not included.patch that fixes the issue in EAP 5.0.0 and EAP 5.0.1 or upgrade to  JBoss EAP 5.1.0 where the patch is already incorporated.  If this issue is caused by symlinks for deployments themselves, don't set the forceCanonical flag as indicated by the patch instructions.

NOTE: Using symlinks in the JBoss directory hierarchy is not recommended in EAP 5.x.x.

Root Cause

The problem is caused by two issues. The primary issue is that the cached deployment is not found in the Permanent Roots cache. The cache used is the CombinedVFSCache which is configured in $JBOSS_HOME/server/$PROFILE/conf/bootstrap/vfs.xml. In this configuration, several permanentroots are configured. This indicate areas that are cached permanently in the CombinedVFSCache:

<property name="permanentRoots">
  <map keyClass="java.net.URL" valueClass="org.jboss.virtual.spi.ExceptionHandler">
    <entry>
      <key>${jboss.lib.url}</key>
      <value><null/></value>
    </entry>
    <entry>
      <key>${jboss.common.lib.url}</key>
      <value><inject bean="VfsNamesExceptionHandler"/></value>
    </entry>
    <entry>
      <key>${jboss.server.lib.url}</key>
      <value><inject bean="VfsNamesExceptionHandler"/></value>
    </entry>
    <entry>
      <key>${jboss.server.home.url}deploy</key>
      <value><inject bean="VfsNamesExceptionHandler"/></value>
    </entry>
    <!-- Only relevant for "all" and "production" server configurations -->
    <entry>
      <key>${jboss.server.home.url}farm</key>
      <value><inject bean="VfsNamesExceptionHandler"/></value>
    </entry>
   </map>
</property>

The crux of the issue is that the Content from anonsvn.jboss.org is not included.URLEditor PropertyEditor that converts the string in this configuration to a URL calls File#getCanonicalFile() (this occurs in the makeURLFromFilespec() method of the Content from anonsvn.jboss.org is not included.Strings utility class.) The getCanonicalFile() method resolves the symlink. So, the permanent root is stored by the canonical path while checks against it use the path containing the symlink and never match. (The second work-around fixes this by causing all paths to use the resolved version.)

At this point, the "non-permanent" cache is used which causes the duplication of the temporary files. (See Content from jira.jboss.org is not included.JBVFS-134 and Content from jira.jboss.org is not included.JBAS-7728.)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.