MongoDB ops manager "java.lang.OutOfMemoryError: unable to create native thread"

Question

Im currently setting up a new MongoDB ops manager machine. Installation works fine but I can't start the mongodb-mms service. The starting of Instance 0 fails with an java.lang.OutOfMemoryError exception. I use the same configuration as on my test server (2 CPU cores, 8gb ram), there the service starts without any interrupt. Changing the ulimit configuration / starting the service with root user has no effect.

New Server specs:

10 Vcores at 2.0Ghz
48gb Ram
800gb storage
Ubuntu 18.04 LTS 64bit

Since the new server is shared with others is it possible that the host limited the cpu usage per user?

mms0.log:

[Starting Logging - App Version: 4.2.23.57072.20210126T1748Z]
2021-03-28T19:32:11.682+0000 [main] INFO  com.xgen.svc.mms.dao.mongo.MongoSvcUriImpl [MongoSvcUriImpl.java.initMorphiaMapper:154] - Initialized Morphia in 12538ms
2021-03-28T19:32:12.319+0000 [main] INFO  com.xgen.svc.mms.dao.mongo.MongoSvcUriImpl [MongoSvcUriImpl.java.<init>:89] - Created MongoSvc with 1 client(s)
[Starting Logging - App Version: 4.2.23.57072.20210126T1748Z]
2021-03-28T19:33:07.998+0000 [main] INFO  com.xgen.svc.core.ServerMain [ServerMain.java.doPreFlightCheck:295] - Starting pre-flight checks
2021-03-28T19:33:20.990+0000 [main] INFO  com.xgen.svc.mms.dao.mongo.MongoSvcUriImpl [MongoSvcUriImpl.java.initMorphiaMapper:154] - Initialized Morphia in 12920ms
2021-03-28T19:33:21.555+0000 [main] INFO  com.xgen.svc.mms.dao.mongo.MongoSvcUriImpl [MongoSvcUriImpl.java.<init>:89] - Created MongoSvc with 1 client(s)
2021-03-28T19:33:22.983+0000 [main] INFO  com.xgen.svc.core.ServerMain [ServerMain.java.doPreFlightCheck:301] - Successfully finished pre-flight checks
2021-03-28T19:33:22.984+0000 [main] INFO  com.xgen.svc.core.ServerMain [ServerMain.java.start:308] - Starting mms...
2021-03-28T19:33:23.142+0000 [main] INFO  com.xgen.svc.core.ServerMain [ServerMain.java.createNonSSLConnector:843] - Creating HTTP listener on *:8080
2021-03-28T19:33:23.438+0000 [main] ERROR com.xgen.svc.core.ServerMain [ServerMain.java.main:226] - Cannot start mms server [FATAL-EXITING] - instance: 0  - msg: unable to create native thread: possibly out of memory or process/resource limits reached
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:803)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:660)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.ensureThreads(QueuedThreadPool.java:642)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.doStart(QueuedThreadPool.java:182)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at com.xgen.svc.core.ServerMain.start(ServerMain.java:424)
        at com.xgen.svc.core.ServerMain.main(ServerMain.java:221)

mms0-startup.log

[23,180s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 512k, guardsize: 0k, detached.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[19,947s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 512k, guardsize: 0k, detached.
Cannot start mms server [FATAL-EXITING] - instance: 0  - msg: unable to create native thread: possibly out of memory or process/resource limits                                                                                              reached
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:803)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:660)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.ensureThreads(QueuedThreadPool.java:642)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.doStart(QueuedThreadPool.java:182)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at com.xgen.svc.core.ServerMain.start(ServerMain.java:424)
        at com.xgen.svc.core.ServerMain.main(ServerMain.java:221)

ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1544321
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62987
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

paulsm4 · Accepted Answer · 2021-03-29 23:08:27Z

1

SUGGESTION: focus on your JVM;

Ensure you have a 64-bit version of Java
Try tuning your JVM parameters:

https://docs.opsmanager.mongodb.com/current/reference/troubleshooting/system/
Open mms.conf in your preferred text editor.
Find this line:
JAVA_MMS_UI_OPTS="${JAVA_MMS_UI_OPTS} -Xss228k -Xmx4352m -Xms4352m -XX:NewSize=600m -Xmn1500m -XX:ReservedCodeCacheSize=128m -XX:-OmitStackTraceInFastThrow"
Change the -Xmx and -Xms values to a larger value. Both parameters should be set to the same value to remove any performance impact from the VM constantly reclaiming memory from the heap.
The value is specified as #k|m|g: a number followed by

k (kilobytes), m (megabytes), or g (gigabytes)

By default, Xmsx and Xms are both set to 4,352 MB (4352m).

EXAMPLE: To set the Java heap to 10 GB, set this value to:
-Xmx10g -Xms10g

STRONG SUGGESTION: I would continue to focus on JVM settings, However, this link might also be relevant:

https://stackoverflow.com/a/31445282/421195

I encountered a similar issue in our Test Ops Manager deployment when we upgraded to Ops Manager 1.8.0. I ultimately opened up a ticket with MongoDB Support and this was the resolution for our issue:

The Ops Manager components are launched using the default username "mongodb-mms". Please adjust the ulimit settings for this user to match those of the "mongodb" user, currently defined in /etc/security/limits.d/99-mongodb-mms-automation-agent.conf.

You may wish to add a separate file under /etc/security/limits.d/ for the mongodb-mms user.

More information can be found here.

New information:

So I tried a fresh install with the same version of MongoDB (4.4.3) and Ops Manager(4.4.8.100) to check if something was wrong with the newest versions. Throws the same error.

I tried running jconsole -debug ->
[1,323s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached

This suggests you might be running out of threads.

Relevant links:

https://github.com/elastic/elasticsearch/issues/31982

Elasticsearch version (bin/elasticsearch --version): 6.3.1

JVM version (java -version):10

OS version:centos

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached but my os has free 80g memory

i used docker.elastic.co/elasticsearch/elasticsearch:6.3.1,
jvm config:
-Xms32g
-Xmx32g
...

[I had] a similar (but likely unrelated) issue in our app (which is using the ES client). For whatever reason, it had gone berserk during the weekend, spawning 9400 threads which made the machine fail in new thread creation for the same user account.

ps -o nlwp,pid -fe helped me spot this, so I could kill the bad process and get the system back to a usable state. Greatly appreciated!

Here is an example ps -o nlwp,pid -fe from my Ubuntu system (an AWS VM). I suspect your "ps" will look very, very different:

# ps -o nlwp,pid -fe
NLWP   PID
   1 13409
   1 13410
   1 13418
   1   915
   1   911

Addendum:

I switched the OS (from Ubuntu 18.04 LTS 64bit) to CentOS 8 and now its working perfectly.

edited Mar 29, 2021 at 23:08

answered Mar 28, 2021 at 20:05

paulsm4

123k23 gold badges175 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

OberDulli94 Over a year ago

Thanks for the answer, I'm currently running this versions of java: openjdk version "11.0.10" 2021-01-19 OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.18.04) OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.18.04, mixed mode, sharing). I also changed the JAVA_MMS_UI_OPTS in the mms.conf file. Doens't do anything.

paulsm4 Over a year ago

Please continue trying to troubleshoot from a Java perspective. Please update your post with the NEW values you set for JAVA_MMS_UI_OPTS, and please CONFIRM the configuration values actually took effect (e.g. using a JMX-aware tool like jconsole). You can also use tools like jconsole or VisualVM to troubleshoot in more detail.

OberDulli94 Over a year ago

So I tried a fresh install with the same version of MongoDB (4.4.3) and Ops Manager(4.4.8.100) to check if something was wrong with the newest versions. Thorws the same error. Now I get an new error when I'm starting jconsole with jconsole -debug <pid> ->

[1,323s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached

. So I guess youre right and the problem is caused by the JVM. Kinda weird is that I can run a stress test ( stress --cpu 10 --io 4 --vm 8 --vm-bytes 8192M --timeout 60s) without any Interrupts.

paulsm4 Over a year ago

Actually, if you're getting similar errors with jconsole ... perhaps we should revisit the OS environment. Specifically 1) #/threads, 2) whether the configuration limit is a) OS-wide, b) user-specific ... or c) specific to some particular cgroup. Run ps -o nlwp,pid -fe. Look at stackoverflow.com/a/47082934/421195 and github.com/elastic/elasticsearch/issues/31982. See my updates above.

OberDulli94 Over a year ago

ps -o nlwp,pid -fe prints [i.ibb.co/T0KxJTx/grafik.png] . When I try to check the kernel-max values (eg. threads, pid) I get an no permission error (/proc/sys/kernel/threads-max: No Permission). I'm logged in as root and the files got -r--r--r-- settings.

|

Collectives™ on Stack Overflow

MongoDB ops manager "java.lang.OutOfMemoryError: unable to create native thread"

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related