OpenBSD malloc options

April 23, 2024

TL;DR: On OpenBSD, while testing, set MALLOC_OPTIONS="SCFGJRU" in your environment.

Linux, Windows, and even FreeBSD have C compilers which provide sanitizers to help catch common errors with memory allocation (use-after-free, buffer overflows etc), and undefined behaviour (signed integer overflow, for example). These are often useful to catch silly mistakes quickly, and address sanitization even comes with hooks to instrument custom memory allocators.

I’ve been working a lot on OpenBSD for the last year or so. OpenBSD doesn’t support sanitizers, and every now and then when I run my code on linux the sanitizers catch one or two issues which I’d liked to have caught before. Recently, while working on one of my projects (a new optimizing backend for my compiler) I caused a program failure due to some sort of memory bug (which OpenBSD reported) but not reproducibly. The actual bug here was minor and in “first-pass” code which was due a tidy up which would have probably ended up using a more robust and well-tested implementation of the same code, but I wanted to try and pin the failure more robustly. The OpenBSD developers have a reputation for writing robust and secure C code, so how do they catch these things? This led me to look into OpenBSD’s malloc options.

OpenBSD lets you configure malloc to help detect common memory issues. These can be set system wide (with a sysctl option), explicitly withing a program with e.g.

extern char *malloc_options;
malloc_options = "S";

or, more usefully for debugging existing builds of applications, by setting an environment variable (MALLOC_OPTIONS).

If you are on OpenBSD, man malloc provides some instructions on what the options are, and what they do for you. The general idea is you set MALLOC_OPTIONS to some string of letters (which have various meanings) - upper case letters turn options on, and the corresponding lower case letters turn them off.

These are the useful ones for our purposes (OpenBSD 7.3):

S - turns on “all options suitable for security auditing”. It’s unclear what these are (I’ve yet to investigate), but it sounds good, so probably turn this on.
C - adds canaries to the end of allocations to detect heap overflows when freeing memory - every now and then I typo a size calculcation, so this one is useful.
F enable more extensive double free and use-after-free detection. I rarely make these sorts of mistakes, at least on the things I’ve written in the last few years, but reassuring to have this turned on for testing.
G guard pages - each allocated block of pages is terminated with a guard page. Reads or writes to the guard page cause a fault at the read or write - so even though this is coarser than canaries, it’s worth turning this on.
J increase junking - writes predictable patterns into allocated and freed memory. Helps diagnose use-before-initialize and use-after-free bugs.
R always realloc - this forces all realloc calls to relocate memory. This is useful for catching pointers which aren’t updated when memory is reallocated (they may appear to work for a while if the existing regions right-hand-end can just be extended).
U use-after-free protection for larger allocations - free pages are protected to cause faults on access.

There are a few more to look at, but these seem like the ones to have turned on while building and testing software.

Perhaps S covers the lot, but I now use MALLOC_OPTIONS="SCFGJRU" when running to help catch memory issues. This made my issue reproducible, and let me track down the problem!

Address sanitizer (and sanitizers in general) are still very useful, and I probably prefer them for every day use. malloc options help catch a slightly different set of issues though, and in the absence of sanitizers are worth turning on. It’s also worth noting that most of the options relate to quite simple tools which are easy to replicate in custom allocators / platform layers.