Analyzing OutOfMemoryError: unable to create new native thread

In your java application if you had seen below error in logs

SOURCE:java.lang.OutOfMemoryError: unable to create new native thread

This means that when java process wants to create a new thread and requested for new process to kernel, but kernel was not able to allocate new process because of the number of processes created by that user reached current limits on max user processes.

You can find the current limits of the user, by running below command

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15244
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

If you check above max user processes which is set to 1024, meaning current user can create 1024 processes, if current processes count exceeds this limit and if any application asking for new process kernel will not allocate new process.

Note there is also a max limit at system level, this we can find from

$ sudo sysctl -a | grep kernel.pid_max
kernel.pid_max = 32768

This means system wide for all users kernel can create 32768 processes.

It is up to the process on how to react when this happened. In case of java we will see SOURCE:java.lang.OutOfMemoryError: unable to create new native thread

You can also find the average load of number of processes created.

$ more /proc/loadavg
7.35 7.83 7.56 6/7802 5393

in above you can see that on an average 7802 processes are getting created.

How to know the current number of processes running for the current user?

$ ps -ef | grep ^app | awk '{print $2}' | grep -v PID | xargs -i sh -c 'ps -p {} -lfT | grep -v PID | wc -l' | awk '{sum+=$1} END {print sum}'
7335

above command searches for all process created by app user and for each process it finds the count of all sub process/threads and sums up all the counts. In above case 7335 processes are currently running for that user app .

You can use a sample program like below to create threads and verify at what point java process will get OutOfMemoryError

Note: Above creates 775 threads, you can increase this number

To fix this, if your system has enough memory, then you can increase the user processes limit. ulimit -u 16384

To make it permanent, you include in ~/.bash_profile

$ more ~/.bash_profile
ulimit -u 16384

or in in /etc/security/limits.conf add below lines

app           soft    nproc          16384
app           hard    nproc          16384

In order to reflect above changes, the java process should be restarted.

How do we know if above limits got applied?

You can find the limits of all the java processes running in that system by below command.

$ ps -ef | grep "java" | grep -v grep | awk '{print $2}' | grep -v PID | xargs -i sh -c 'echo {}; cat /proc/{}/limits | grep "Max processes"'

Max processes             8192                 8192                 processes
59525
Max processes             8192                 8192                 processes
60232
Max processes             16384                16384                processes

in above 60232 only got the new limits, remaining 2 java process needs to be restarted.

Hope this helps.