Set python memory limit on Ubuntu 12.04

Question

Currently my script runs an algorithm that require allocation of many objects, as a result I receive a "Memory Error".

However, from monitoring my memory usage, the error occurs when I'm only using 4Gb out of 16Gb of my total memory capabilities on my computer. Is there a way to increase the allowed memory consumption for my python script up to 16Gb?

I have tried to use https://docs.python.org/2/library/resource.html#resource.setrlimit and set everything to MAX_VALUE, yet I still have the error at 4Gb of memory consumption.

Any pointers will be helpful.

I have read: Set Python memory limit and many posts like it but they don't directly address my problem.

Here is a what "ulimit -a" outputs, if it is of any use.

evan@evan-box:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 96626
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 96626
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

thanks.

Denilson Sá Maia · Accepted Answer · 2014-08-05 18:56:23Z

1

Are you running a 64bit kernel check the following

file /sbin/init

uname -m

edited Aug 5, 2014 at 18:56

Denilson Sá Maia

49.9k35 gold badges115 silver badges115 bronze badges

answered Aug 5, 2014 at 18:16

wwright

3642 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Evan Pu Over a year ago

evan@evan-box:~$ file /sbin/init /sbin/init: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x07075fcb55b05aeb6286efabba63534fa6ecd213, stripped evan@evan-box:~$ uname -m i686

Evan Pu Over a year ago

so actually with a 32bit kernel I can't even use all of my memory?! gasp

Denilson Sá Maia Over a year ago

2**32 bytes is 4 GiB. 32-bit executables can't address more than 4 GiB of RAM. Please make sure your system (both kernel and the python interpreter) are 64-bit.

wwright Over a year ago

if you are using a VM you could just create another VM and use a 64b install

Collectives™ on Stack Overflow

Set python memory limit on Ubuntu 12.04

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related