Memory leaks? No, thanks.

By: Carsten Emde

Did you know that a raven can help?

Regular visitors of the OSADL web site and of its QA Farm already noted that we have a certain fervent enthusiasm for Munin. Munin is such a great tool; there probably are very few problems in the world that Munin cannot solve or at least help solving. The name Munin, by the way, is derived from the name of one of the two ravens that are, according to Norse mythology, sitting on God Odin's shoulders and whispering news in his ears when back from flying over the country. So does Munin today, it flies over your computerland and returns bad and good news from the devices.

Munin can also help to diagnose memory leaks - the greatest enemy of programmers who develop systems that normally never are switched off. Since such systems – for example servers and embedded systems – often provide crucial functionality, a crash caused by a memory leak may have fatal consequences but certainly is no a fateful event. In contrast, it could have been predicted easily.

Ok, so how is Munin used for this purpose? Simply have a look at the memory usage plot and let the alarm bells ring, if you see a cuneiform shape of one of the memory components. The only exceptions are, of course, caches, since they typically grow until they occupy the entire memory that is not used otherwise. A typical deleterious short-term memory leak is given in the center part of Figure 1.

Figure 1: A fatal short-term application memory leak

However, Munin has proven to be much more helpful when a slow memory leak is eating your memory. Such slow leaks may not be obvious even during a longer system test of, let's say, two weeks. A two-week memory recording with Munin, however, may already be very typical for a memory leak as exemplified in the somewhat historical recording from 2007 in Figure 2. The cuneiform shape of the memory leak is, of course, better visible in a compressed time scale when an entire year is displayed.

Figure 2: Slow application memory leaks

Memory leaks may not always directly let the system crash. If swap space is available, the system will first become very slow and even inaccessible because of the high amount of memory swaps before it finally crashes (Figure 3).

 Figure 3: Effect of a memory leak on swap activity

But we should not finish this article without showing an example of a normal recording that gives no suspicion at all of a memory leak (Figure 4). After all, this is by far the most frequent finding. Note that the slight increase of the slab cache that consumes some of the memory cache is normal and is no cause for any concern.

 Figure 4: Normal recording, no signs of a memory leak.

Recommendation

If you want to be sure that your system does not suffer from a memory leak that may lead to rare, obscure and absolutely unacceptable crashes, install Munin and display the course of the memory usage. If you want this to be done in a professional and inexpensive way, checkout the OSADL QA services and hire rack space at the OSADL QA Farm! OSADL members even have a number of systems free according to their membership level.

---