0

I have an embedded device running a C/C++ application on Linux. There is an issue with the software that is causing the application to hog memory until the Linux OS kills the app.

Below is the output from the kernel when the app dies.

Feb  2 05:35:23 myEmbeddedDevice kernel: myApplicaion invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Feb  2 05:35:24 myEmbeddedDevice kernel: myApplicaion cpuset=/ mems_allowed=0
Feb  2 05:35:24 myEmbeddedDevice kernel: CPU: 1 PID: 949 Comm: myApplicaion Tainted: G           O 3.12.0 #1
Feb  2 05:35:24 myEmbeddedDevice kernel: [<80016394>] (unwind_backtrace+0x0/0xf4) from [<80011de0>] (show_stack+0x10/0x14)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<80011de0>] (show_stack+0x10/0x14) from [<80404d6c>] (dump_stack+0x80/0x90)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<80404d6c>] (dump_stack+0x80/0x90) from [<80403344>] (dump_header.isra.13+0x84/0x190)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<80403344>] (dump_header.isra.13+0x84/0x190) from [<800a9210>] (oom_kill_process+0x278/0x3dc)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800a9210>] (oom_kill_process+0x278/0x3dc) from [<800a97f8>] (out_of_memory+0x29c/0x2c0)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800a97f8>] (out_of_memory+0x29c/0x2c0) from [<800ad670>] (__alloc_pages_nodemask+0x86c/0x8c8)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800ad670>] (__alloc_pages_nodemask+0x86c/0x8c8) from [<800a8254>] (filemap_fault+0x1cc/0x420)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800a8254>] (filemap_fault+0x1cc/0x420) from [<800c57bc>] (__do_fault+0x68/0x498)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800c57bc>] (__do_fault+0x68/0x498) from [<800c877c>] (handle_mm_fault+0xec/0x800)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800c877c>] (handle_mm_fault+0xec/0x800) from [<8001adc8>] (do_page_fault+0x1d4/0x3b4)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<8001adc8>] (do_page_fault+0x1d4/0x3b4) from [<800083e8>] (do_DataAbort+0x38/0x9c)
Feb  2 05:35:24 myEmbeddedDevice kernel: [<800083e8>] (do_DataAbort+0x38/0x9c) from [<80012a34>] (__dabt_usr+0x34/0x40)
Feb  2 05:35:24 myEmbeddedDevice kernel: Exception stack(0x8fb51fb0 to 0x8fb51ff8)
Feb  2 05:35:24 myEmbeddedDevice kernel: 1fa0:                                     3fcfffff 76e19390 76e19398 76e04170
Feb  2 05:35:24 myEmbeddedDevice kernel: 1fc0: 548fea44 00000800 00000010 00000009 0188c068 00002000 00000400 00000010
Feb  2 05:35:24 myEmbeddedDevice kernel: 1fe0: 3fc921fb 548fea40 76dcf440 76dcf4b0 80000010 ffffffff
Feb  2 05:35:24 myEmbeddedDevice kernel: Mem-info:
Feb  2 05:35:24 myEmbeddedDevice kernel: Normal per-cpu:
Feb  2 05:35:24 myEmbeddedDevice kernel: CPU    0: hi:   90, btch:  15 usd:  55
Feb  2 05:35:24 myEmbeddedDevice kernel: CPU    1: hi:   90, btch:  15 usd:  77
Feb  2 05:35:24 myEmbeddedDevice kernel: active_anon:60356 inactive_anon:10 isolated_anon:0
Feb  2 05:35:24 myEmbeddedDevice kernel:  active_file:37 inactive_file:62 isolated_file:0
Feb  2 05:35:24 myEmbeddedDevice kernel:  unevictable:0 dirty:2 writeback:0 unstable:0
Feb  2 05:35:24 myEmbeddedDevice kernel:  free:522 slab_reclaimable:487 slab_unreclaimable:829
Feb  2 05:35:24 myEmbeddedDevice kernel:  mapped:46 shmem:39 pagetables:456 bounce:0
Feb  2 05:35:24 myEmbeddedDevice kernel:  free_cma:0
Feb  2 05:35:24 myEmbeddedDevice kernel: Normal free:2088kB min:2012kB low:2512kB high:3016kB active_anon:241424kB inactive_anon:40kB active_file:128kB inactive_file:356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:253604kB mlocked:0kB dirty:8kB writeback:0kB mapped:132kB shmem:156kB slab_reclaimable:1948kB slab_unreclaimable:3316kB kernel_stack:480kB pagetables:1824kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1450 all_unreclaimable? yes
Feb  2 05:35:24 myEmbeddedDevice kernel: lowmem_reserve[]: 0 0
Feb  2 05:35:24 myEmbeddedDevice kernel: Normal: 33*4kB (EMR) 12*8kB (EMR) 5*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 2292kB
Feb  2 05:35:24 myEmbeddedDevice kernel: 116 total pagecache pages
Feb  2 05:35:24 myEmbeddedDevice kernel: 0 pages in swap cache
Feb  2 05:35:24 myEmbeddedDevice kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb  2 05:35:24 myEmbeddedDevice kernel: Free swap  = 0kB
Feb  2 05:35:24 myEmbeddedDevice kernel: Total swap = 0kB
Feb  2 05:35:24 myEmbeddedDevice kernel: 65536 pages of RAM
Feb  2 05:35:24 myEmbeddedDevice kernel: 799 free pages
Feb  2 05:35:24 myEmbeddedDevice kernel: 2135 reserved pages
Feb  2 05:35:24 myEmbeddedDevice kernel: 971 slab pages
Feb  2 05:35:24 myEmbeddedDevice kernel: 30 pages shared
Feb  2 05:35:24 myEmbeddedDevice kernel: 0 pages swap cached
Feb  2 05:35:24 myEmbeddedDevice kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Feb  2 05:35:24 myEmbeddedDevice kernel: [  861]     0   861      557       40       4        0             0 rpcbind
Feb  2 05:35:24 myEmbeddedDevice kernel: [  878]     0   878      465       21       4        0             0 atd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  888]   999   888      653       35       4        0             0 dbus-daemon
Feb  2 05:35:24 myEmbeddedDevice kernel: [  895]     0   895     1039       66       4        0         -1000 sshd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  920]     0   920      652       48       4        0             0 rc
Feb  2 05:35:24 myEmbeddedDevice kernel: [  921]     0   921   203804    59132     385        0             0 myApplicaion
Feb  2 05:35:24 myEmbeddedDevice kernel: [  923]     0   923      421       15       3        0             0 logger
Feb  2 05:35:24 myEmbeddedDevice kernel: [  928]     0   928      445       38       3        0             0 syslogd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  930]     0   930      760      369       4        0             0 klogd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  935]     0   935      572       38       4        0             0 lighttpd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  938]     0   938      618      116       3        0             0 crond
Feb  2 05:35:24 myEmbeddedDevice kernel: [  944]     0   944      550       14       3        0             0 getty
Feb  2 05:35:24 myEmbeddedDevice kernel: [  945]     0   945      550       14       3        0             0 getty
Feb  2 05:35:24 myEmbeddedDevice kernel: [  950]     0   950     1100       88       5        0             0 sshd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  952]     0   952      717       77       4        0             0 sh
Feb  2 05:35:24 myEmbeddedDevice kernel: [  953]     0   953      553       48       4        0             0 top
Feb  2 05:35:24 myEmbeddedDevice kernel: [  954]     0   954     1100       88       5        0             0 sshd
Feb  2 05:35:24 myEmbeddedDevice kernel: [  956]     0   956      717       77       4        0             0 sh
Feb  2 05:35:24 myEmbeddedDevice kernel: [  957]     0   957      431       17       3        0             0 tail
Feb  2 05:35:24 myEmbeddedDevice kernel: Out of memory: Kill process 921 (myApplicaion) score 909 or sacrifice child
Feb  2 05:35:24 myEmbeddedDevice kernel: Killed process 921 (myApplicaion) total-vm:815216kB, anon-rss:236528kB, file-rss:0kB

Can I use this information to debug my code? Which information in the errors is most useful to get started in trying to figure out why the app is leaking?

1 Answer 1

1

No, it won't help you much. myApplication had consumed nearly 60MB of RAM (of the 64MB available on your device) when the OOM killer struck it down.

You will have to look into the myApplication process itself.

Depending on your actual platform/toolchain you have various options (adress-sanitizer, valgrind, electric-fence, ...).

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.