Background
We are developing a Java service in Heroku with 1 dyno, which is using Memcached cloud.
Issue
Meanwhile we were developing and testing it, It was working fine. However, when we decided to test it in a real environment, It started to return the following error:
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.Thread.start0(Native Method)
java.lang.Thread.start(Thread.java:717)
net.spy.memcached.MemcachedConnection.<init>(MemcachedConnection.java:306) net.spy.memcached.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:209)
net.spy.memcached.MemcachedClient.<init>(MemcachedClient.java:209)
Memcached.<init>(Memcached.java:34)
Main.lambda$main$1(Main.java:101)
spark.SparkBase$1.handle(SparkBase.java:311)
spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:159)
spark.webserver.JettyHandler.doHandle(JettyHandler.java:60)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:179)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
org.eclipse.jetty.server.Server.handle(Server.java:451)
org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:252)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:266)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:240)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527)
java.lang.Thread.run(Thread.java:748)
We didn't have so much clients to arrive at 256 process or threads (limit of 1 dyno)
Workaround
We don't know it yet
Code
<!--Main-->
//...
try {
memcached = new Memcached ();
}
catch (Exception e) {
memcached = null;
}
//...
<!--Memcached-->
public class Memcached {
private MemcachedClient memcachedClient;
private boolean connected;
public Memcached () throws Exception {
setConnected(true);
try {
AuthDescriptor ad = new AuthDescriptor(new String[] { "PLAIN" },
new PlainCallbackHandler(System.getenv("MEMCACHEDCLOUD_USERNAME"),
System.getenv("MEMCACHEDCLOUD_PASSWORD")
)
);
setMemcachedClient(new MemcachedClient(
new ConnectionFactoryBuilder()
.setDaemon(true)
.setFailureMode(FailureMode.Retry)
.setProtocol(ConnectionFactoryBuilder.Protocol.BINARY) //this is the line 34
.setAuthDescriptor(ad).build(),
AddrUtil.getAddresses(System.getenv("MEMCACHEDCLOUD_SERVERS"))
));
} catch (Exception ex) {
// the Memcached client could not be initialized
setConnected(false);
throw new Exception ("{\"ErrorCode\":20,\"Portal\":\"Memcached\",\"ResponseCode\":\"\",\"Message\":\""
+ ex.getMessage() + "\"}");
}
}
public String get (String service, String fromCurrency, String toCurrency) {
if (!isConnected()) {
return null;
}
try {
return (String) getMemcachedClient().get(service + ";" + fromCurrency + ";" + toCurrency);
}
catch (Exception e) {
return null;
}
}
public boolean set (String service, String fromCurrency, String toCurrency, String value) {
if (!isConnected()) {
return false;
}
try {
double hour = Double.parseDouble(System.getenv("TIME_CACHE_HOUR"));
getMemcachedClient().set(service + ";" + fromCurrency + ";" + toCurrency, (int)(60 * 60 * hour), value);
return true;
}catch (Exception e) {
return false;
}
}
public boolean isConnected () {
return this.connected;
}
private void setConnected (boolean connected) {
this.connected = connected;
}
private MemcachedClient getMemcachedClient () {
return this.memcachedClient;
}
private void setMemcachedClient (MemcachedClient memcachedClient) {
this.memcachedClient = memcachedClient;
}
Anyone knows how can we fix this error?
EDIT: I have read the post Java: Unable to create new native thread, but It isn't our problem. We checked the attributes that they said there, but we have good values. However, I could see that everytime that memcached is initialized, It create a new thread, but that thread isn't deleted when the service stop. There's the problem.
