16

Background

I am developing an Android App which provides a simple HTTP/HTTPS server. If the HTTPS serving is configured then on every connection an increasing native memory usage is observed which eventually leads to an app crash (oom), while using the HTTP configuration keeps the native memory usage relative constant. The app's Java VM keeps relative constant in both configurations.

The app serves an HTML page which contains a javascript with periodic polling (one json poll every second), so calling the app page using the HTTPS configuration and keeping the page open for several hours will lead to the mentioned out-of-memory because of increasing native memory usage. I have tested many SSLServerSocket and SSLContext configurations found on internet with no luck.

I observe the same problem on various Android devices and various Android versions beginning with 2.2 up to 4.3.

The code for handling client requests is the same for both configurations HTTP/HTTPS. The only difference between the two configurations is the setup of the server socket. While in the case of HTTP server socket one single line similar to this "ServerSocket serversocket = new ServerSocket(myport);" does the job, in the case of HTTPS server setup the usual steps for setting up the SSLContext are taken -- i.e. setting up the keymanager and initializing the SSLContext. For now, I use the default TrustManager.

Need For Your Advice

Does somebody know about any memory leak problems in Android's default TLS Provider using OpenSSL? Is there something special I should consider to avoid the leak in the native memory? Any hint is highly appreciated.

Update: I have also tried both TLS providers: OpenSSL and JSSE by explicitly giving the provider name in SSLContext.getInstance( "TLS", providerName ). But that did not change anything.

Here is a code block which demonstrates the problem. Just create a sample app put it into the bottom of the main activity's onCreate and build & run the app. Make sure that your Wifi is on and call the HTML page by following address:

https://android device IP:9090

Then watch the adb logs, after a while you will see the native memory beginning to increase.


new Thread(new Runnable() {

public void run() {

final int PORT = 9090; SSLContext sslContext = SSLContext.getInstance( "TLS" ); // JSSE and OpenSSL providers behave the same way KeyManagerFactory kmf = KeyManagerFactory.getInstance( KeyManagerFactory.getDefaultAlgorithm() ); KeyStore ks = KeyStore.getInstance( KeyStore.getDefaultType() ); char[] password = KEYSTORE_PW.toCharArray(); // we assume the keystore is in the app assets InputStream sslKeyStore = getApplicationContext().getResources().openRawResource( R.raw.keystore ); ks.load( sslKeyStore, null ); sslKeyStore.close(); kmf.init( ks, password ); sslContext.init( kmf.getKeyManagers(), null, new SecureRandom() ); ServerSocketFactory ssf = sslContext.getServerSocketFactory(); sslContext.getServerSessionContext().setSessionTimeout(5); try { SSLServerSocket serversocket = ( SSLServerSocket )ssf.createServerSocket(PORT); // alternatively, the plain server socket can be created here //ServerSocket serversocket = new ServerSocket(9090); serversocket.setReceiveBufferSize( 8192 ); int num = 0; long lastnatmem = 0, natmemtotalincrease = 0; while (true) { try { Socket soc = (Socket) serversocket.accept(); Log.i(TAG, "client connected (" + num++ + ")"); soc.setSoTimeout(2000); try { SSLSession session = ((SSLSocket)soc).getSession(); boolean valid = session.isValid(); Log.d(TAG, "session valid: " + valid); OutputStream os = null; InputStream is = null; try { os = soc.getOutputStream(); // just read the complete request from client is = soc.getInputStream(); int c = 0; String itext = ""; while ( (c = is.read() ) > 0 ) { itext += (char)c; if (itext.contains("\r\n\r\n")) // end of request detection break; } //Log.e(TAG, " req: " + itext); } catch (SocketTimeoutException e) { // this can occasionally happen (handshake timeout) Log.d(TAG, "socket timeout: " + e.getMessage()); if (os != null) os.close(); if (is != null) is.close(); soc.close(); continue; } long natmem = Debug.getNativeHeapSize(); long diff = 0; if (lastnatmem != 0) { diff = natmem - lastnatmem; natmemtotalincrease += diff; } lastnatmem = natmem; Log.i(TAG, " answer the request, native memory in use: " + natmem / 1024 + ", diff: " + diff / 1024 + ", total increase: " + natmemtotalincrease / 1024); String html = "<!DOCTYPE html><html><head>"; html += "<script type='text/javascript'>"; html += "function poll() { request(); window.setTimeout(poll, 1000);}\n"; html += "function request() { var xmlHttp = new XMLHttpRequest(); xmlHttp.open( \"GET\", \"/\", false ); xmlHttp.send( null ); return xmlHttp.responseText; }"; html += "</script>"; html += "</head><body onload=\"poll()\"><p>Refresh the site to see the inreasing native memory when using HTTPS: " + natmem + " </p></body></html> "; byte[] buffer = html.getBytes("UTF-8"); PrintWriter pw = new PrintWriter( os ); pw.print("HTTP/1.0 200 OK \r\n"); pw.print("Content-Type: text/html\r\n"); pw.print("Content-Length: " + buffer.length + "\r\n"); pw.print("\r\n"); pw.flush(); os.write(buffer); os.flush(); os.close(); } catch (IOException e) { e.printStackTrace(); } soc.close(); } catch (IOException e) { e.printStackTrace(); } } } catch (SocketException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } }).start();

-- EDIT --

I have uploaded a sample app project called SSLTest for eClipse which demonstrates the problem:

http://code.google.com/p/android/issues/detail?id=59536

-- UPDATE --

Good news: today the reported Android issue above was identified and proper submissions were made to fix the memory leak. For more details see the link above.

3
  • slideshare.net/zblair/… .. brute force analyse your memory usage by ssl. Commented Sep 6, 2013 at 23:15
  • @Robert: I have already analyzed the app memory activities extensively using MAT, the GC does its job properly and I see no source of leaking coming from dangling java object references. The 'native' memory increase seems to have to do with internals of SSL implementation. Just test it yourself by creating an example app as explained above and see yourself. Commented Sep 8, 2013 at 10:00
  • Good news about getting the leak fixed upstream. Thanks for making Android better :) Commented Sep 11, 2013 at 0:23

3 Answers 3

3
+100

I imagine this would be a substantial time investment, but I see that Valgrind has been ported to Android. You could try getting that up and running. Of course, if you find there's an internal memory leak, there isn't a lot you can do about it except attempt to get the bug fixed in future Android releases.

As a workaround, you could make your application multi-process and put the https service in a separate process. That way you could restart it periodically, avoiding OOM. You might also have to have a third process just accepting port 443 connections and passing them on to the https worker - in order to avoid tiny outages when the https worker is restarted.

This also sounds like a substantial time investment :) But it would presumably successfully avoid the problem.

--- EDIT: More detail ---

Yes, if you have a main application with its own UI, a worker process for handling SSL and a worker process for accepting the SSL requests (which as you say probably can't be 443), then on top of your normal Activity classes, you would have two Service classes, and the manifest would place them in separate processes.

Handling SSL process: Rather than waiting for an OOM to crash the service, the service could monitor its own Debug.getNativeHeapSize(), and explicitly restart the service when it increased too much. Either that, or restart automatically after every 100 requests or so.

Handling listening socket process: This service would just listen on the TCP port you choose and pass on the raw data to the SSL process. This bit needs some thought, but the most obvious solution is to just have the SSL process listen on a different local port X (or switch between a selection of different ports), and the listening socket process would forward data to port X. The reason for having the listening socket process is to gracefully handle the possibility that X is down - as it might be whenever you restart it.

If your requirements allow for there being occasional mini-outages I would just do the handling SSL process, and skip the listening socket process, it's a relatively simple solution then - not that different to what you'd do normally. It's the listening socket process that adds complexity to the solution...

Sign up to request clarification or add additional context in comments.

4 Comments

I assume "putting the server in a separate process" means putting the HTTP-Server into an own service. This would mean that the main app would have to deal with service crashes and also with login-recovery as users may be logged in while the crash happens (btw: I think in a non-rooted Android device you cannot serve on port 443, it is reserved for the system and you would need admin permissions for using it). I give you a +1 for your suggestions, though ;-)
By the way, you will probably find - whatever you do - that you will have to handle your service restarting and thus persist logged in users sessions. Android does try and keep Services running for you, but that doesn't mean it's above killing them when it needs resources - it just means that when it kills your service it will attempt to start it up again in a timely manner. However, you can get a better guarantee it won't kill your service if you register it as running in the foreground.
Your suggestions make sense to me and I think separating the serving functionality into an own process may be a good workaround to some limits: 1) when I close the app -- i.e. the onDestroy is called -- then the native memory is not released. The only way to release the native memory is terminating/killing the process which afaik cannot be achieved programmatically, you have to use something like a task or app manager. Please see my link above, I have reported the problem to the Android developers and added an eClipse project which demonstrates the problem. continue ...
2) On some devices, the OOM leads to a device reboot instead of just killing the responsible app. While your suggestions add a level of robustness to the total app functionality, I fear they do not really solve the problem. Again, if you could find some time please build and run the app I have provided in the Android bug report (see my link above). I appreciate your thoughts, hence the bounty is yours my friend ;-)
0

Does it help to explicitly close the input stream? In the sample code the input stream seems to only be closed in the case of a SocketTimeoutException exception.

--EDIT--

You could rename run() to run2() and move the while loop into run() and remove it from run2() and see if that makes a difference? This couldn't be a solution but would tell you if any of the long-lived objects free up the memory when their references are dropped.

2 Comments

No, closing the input stream does not seem to change anything.
Please see my comment to Robert above. The app does not leak any memory because of screwing up the GC. I suspect that the SSL internals cause a memory leak.
0

There is one detail I would recommend changing in your implementation.

Make a list of all your resource variables, for example Sockets, Streams, Writers, etc. Be sure to have the declaration outside your try statement and be sure to do cleanup / closing in the finally statement. I normally do something like this to be 100% sure:

InputStream in = null;
OutputStream out = null;

try {
    //assign a proper value to in and out, and use them as needed.
} catch(IOException e) {
    //normal error handling
} finally {
    try {
        in.close();
    } catch(IOException e) {}
    try {
        out.close();
    } catch(IOException e) {}
}

It looks a little bit confusing, but imagine you use your in Stream inside the try block and you get some Exception, then your Streams never get closed and that is a potential reason for memory leaks.

I cannot guarantee that this is the reason, but it should be a good startup point.

About managing your service. I had a lot of bad experiences with Android services because I was running them in the same thread as the GUI. Under some circumstances, Android will see some code that is executing for too long and kill your main process in order to protect from crashes. The solution I found was to follow the suggestion from this tutorial (look at point 4):

http://www.vogella.com/articles/AndroidServices/article.html

After this, my service just worked as expected and didn't interfere with my GUI Process.

Regards

1 Comment

Have you really tried the code above? The problem is not coming from missing to close any connections. Check the example app I have provided in the Android bug report (see the link I have provided above).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.