Found an interesting wrinkle in the Linux handling of ulimits for maximum memory size and data segment size – they are not enforced by current glibc / kernel configurations in certain conditions.

I tracked this down to the fact that somewhere around glibc 2.3 the malloc() implementation was ripped out and replaced with one that uses mmap() for allocations of 128KB or more. The kicker is that the kernel mmap() implementation only cares about the virtual memory ulimit (RLIMIT_AS) for enforcement, the others are just ignored!

So currently an application which uses small allocations (<128KB) will find malloc() failing when they hit their max mem / data seg size ulimit whereas an application that grabs RAM in larger chunks will sail happily past that without a care in the world..

Bug, feature or undefined behaviour ? You decide.. 🙂