Memories

If there's one thing science has taught me, it's that the best way to sell an idea is with pretty pictures. So, here's a pretty picture:


We've been having memory issues in the game intermittently since we came back, and the diagram offers a good illustration of the problem. When this bug first manifested, it so happened that I was trying to edit something on our game server and was rudely surprised by all manner of warnings about having insufficient memory to even open the text editor. Sure enough, somehow our game's memory footprint had blown up (seemingly overnight) to over twice its normal size, and there was very little to indicate what had happened. The solution then was to immediately reboot the MUD to flush memory and start from scratch, contact our provider and up our memory, and create a tool to keep track of our game's memory usage.

I've been checking the data dumped out by this tool daily to see if the memory blowup could be reproduced, and today was the first day I had a bite on the line. I suspected to find logs of someone doing something stupid coinciding with this big spike (like spamming haggle, exploiting some bug, etc), but there was nothing. Going back to my own personal logs of that morning though, the memory spikes happened around a time when Sidonie and I were working out some problems with a builder command. Ah-ha!

As it turns out, a very useful but not-often-used command that was put into the code many years ago had a very severe memory leak in it. The command uses the regular expression engine to parse boatloads of mobprogs iteratively, but the code wasn't cleaning up the tail end of the regex calls because the regex libraries silently malloc huge blocks of memory that aren't automatically freed when the parent routine exits.

This was a pretty easy mistake to make; while most high-level programming languages in which one would utilize regexes (like Perl, PHP, and whatever else) abstract away all of the memory management, C certainly does not. I guess the pitfall here is that the regcomp() call buried mallocs within it, so a headfirst dive into using regexes in C wouldn't directly expose the programmer to the mallocs which would raise alarms. An easy pitfall, sure, but like a wise man once said, "if you are going to write C code, you better be willing to deal with the memory management."

For what it's worth, I've gotten the game's code to be remarkably scarce on memory leaks like this despite the fact that I (nor any past coder of whom I know) has ever profiled the code with something like Valgrind or even gprof. While this case may give a pretty good argument for trying to figure out a realistic way to do this, blind dependence on tools like Valgrind to make up for incompetent coding has been known to cause serious problems. If we start running into memory leaks again though, maybe I'll try to figure out a way to set up profiling on a test port somewhere and have some of you players beat on it for a night.