Error 'java.net.SocketException Too many open files' when I run a Java application that uses TCP sockets?

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 4.2.x
    • 4.3.x
    • 5.x
    • 6.x
    • 7.x
  • Linux/Solaris

Issue

  • Error 'java.net.SocketException: Too many open files' when I run a Java application that uses TCP sockets?
  • We are getting a too many open files exception on the Jboss EAP server. The below is the entries that we have on sysctl file:
[root@user123 ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key.
  • Also, when we observe the number of open files based on the process id for JBOSS EAP, it is 1082.
  • Let us know if any more inputs are needed to resolve this issue. 
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.forwarding = 0
error: permission denied on key 'net.ipv4.conf.all.mc_forwarding'
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.log_martians = 1
net.ipv4.tcp_max_syn_backlog = 4096

Resolution

  • A java.net.SocketException occurs when there is an underlying problem in the TCP layer of the Operating System. Java passes the error message that the kernel returns and print it along with the exception. In this case, the Java application is running into a situation where it cannot open a new file-handle to be used for a socket. In Java, every socket is a file, so open file limits enforced by the OS will limit the maximum number of sockets.

  • There are 2 instances where the application could be running into a maximum file-handle limit situation:

    • When the number of open files reaches the global limit. This limit is defined in /proc/sys/file-max file.
    • When the number of open files reaches the per user/shell limits. These can be set by ulimit, setrlimit(). By default, the values are set by the Pluggable Authentication Module (PAM).

  • The first place to check would be the global system file-handle limits. This can be done using the sysctl command.

    # /sbin/sysctl fs.file-max
fs.file-max = 82702

  • The value in file-max denotes the maximum number of file-handles that the Linux kernel will allocate. If a java application is getting the "out of files java socket exception" message then this number should be increased. An application that opens a large number of socket descriptors and does not close them, can cause this system limit to be reached. To increase the file-max kernel parameter value, use the sysctl command.

    # /sbin/sysctl -w fs.file-max=<VALUE>

Note: Set the value according to the requirement. In this case, the new value should be greater than the previous value.

  • The second place to check would be the user/shell limits imposed by PAM. These limits are set in the file /etc/security/limits.conf. This file contains limit settings for various items, including the maximum number of files that a user can open. This setting is defined by the item keyword nofile. To check the current settings for a particular user, login to that user's shell and run the command ulimit -n to check the maximum number of open file descriptors for that user.

    # ulimit -n
1024

  • To change this value for the user, who is running the Java application, add the following lines to the /etc/security/limits.conf file:

    <username> soft nofile 2048
<username> hard nofile 2048

  • The user will then be required to login again to the shell and any processes running under that uid will need to be restarted.

  • An important note to mention here is that if the application is being called as part of a cron job, the user limits set by PAM will be ignored since cron does not use a login shell and PAM is bypassed. Please see the following kbase article for a more in depth explanation of how to work around the cron limitations: This content is not included.What can I do to work around the limitation on Red Hat Enterprise 2.1 or 3 regarding PAM limits that do not get applied in cron?

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.