I am writing a C program on linux and I wonder:
How to limit the total memory my c program consumes?
If I set a memory limit to my c program, say 32M, what happens if it requires much more memory than 32M?
You should use the setrlimit system call, with the RLIMIT_DATA and RLIMIT_STACK resources to limit the heap and stack sizes respectively. It is tempting to use RLIMIT_AS or RLIMIT_RSS but you will discover that they do not work reliably on many old Linux kernels, and I see no sign on the kernel mailing list that the problems have been resolved in the latest kernels. One problem relates to how mmap'd memory is counted or rather not counted toward the limit totals. Since glibc malloc uses mmap for large allocations, even programs that don't call mmap directly can exceed the limits.
If you exceed the RLIMIT_STACK limit (call stack too deep, or allocate too many variables on the stack), your program will receive a SIGSEGV. If you try to extend the data segment past the RLIMIT_DATA limit (brk, sbrk or some intermediary like malloc), the attempt will fail. brk or sbrk will return < 0 and malloc will return a null pointer.
rlimit_stack after Stack Clash remediations may result in failure or related problems. Also see Red Hat Issue 1463241On Linux, within your C program, use setrlimit() to set your program's execution environment limits. When you run out of memory, for instance, then calls to malloc() will return NULL etc.
#include <sys/resource.h>
{ struct rlimit rl = {32000, 32000}; setrlimit(RLIMIT_AS, &rl); }
rl is invalid; it assumes a particular ordering of elements. You need to either use C99 designated initializers, or just assign to the elements by name as independent statements.struct rlimit rl = { .rlim_cur = 32000, .rlim_max = 32000 }; ... Also notice the necessity of the struct keyword.