Linux ulimit memory enforcement oddity
Found an interesting wrinkle in the Linux handling of ulimits for maximum memory size and data segment size - they are not enforced by current glibc / kernel configurations in certain conditions.
I tracked this down to the fact that somewhere around glibc 2.3 the malloc() implementation was ripped out and replaced with one that uses mmap() for allocations of 128KB or more. The kicker is that the kernel mmap() implementation only cares about the virtual memory ulimit (RLIMIT_AS) for enforcement, the others are just ignored!
So currently an application which uses small allocations (<128KB) will find malloc() failing when they hit their max mem / data seg size ulimit whereas an application that grabs RAM in larger chunks will sail happily past that without a care in the world..
Bug, feature or undefined behaviour ? You decide.. 
March 11th, 2008 at 12:58 pm
[...] might not be aware of this, but since some time during the glibc 2.3 era, Linux apparently doesn’t enforce the RSS memory limit. (ulimit -m) My tests seem to confirm [...]
March 19th, 2008 at 9:31 am
Thanks Chris for sharing this information with us.