I have an application which has multiple threads and which is constantly allocating and deallocating memory (using malloc/free) After a while (normally days) the virtual memory starts greatly increasing while the number of threads remains constant. I have instrumented the malloc() usage with mallocinfo() and while the arena and allocated memory do increase during this period, the increase in virtual memory is an order of magnitude greater (eg the arena size reported by mallocinfo() is about 300M, but the virtual memory increased from 1.8G to about 2.9G.

When I first saw the virtual memory increasing, I suspected a memory leak caused by a pointer not being freed, but after unless mallocinfo() is not showing all of the malloc'd memory, the increase is happening elsewhere.

There are enough possibilities that you can't do a good job guessing at them. In your situation you should look at the details.

While the process is still running and using significantly more virtual memory than you think is appropriate, grab a copy of the details of its virtual memory use:
Replace pid with the pid of the process you are investigating. Then look through some_file with a text editor to get some understanding of the virtual memory use of the process.

Further investigation depends on what you see in that file. If it is a massive number of tiny allocations, you might need some post processing to dig out the useful info. If you can find some very large individual allocations, those should be directly understandable.

As a first step in understanding one of those files myself, I used a smart search tool in my text editor that let me search for all occurrences of:
Size: followed by 12 ignored characters, followed by a non blank.
That finds every allocation of 100MB of more. In the large process I just looked at there was exactly one allocation over 100MB and that was the bulk of the process's memory use. So it was easy to find that and ignore all the tiny allocations. I don't know whether your example will be that easy. (I don't know how to do such text searches in whatever text editor or viewer you use. But if that is a problem, I'm sure plenty of others at LQ could answer that. I'm on a mixed OS lan, so I used Visual Studio on Windows to look at my file on Linux).

1 members found this post helpful.

I have looked at /proc/pid/smaps both for the application before it starts misbehaving and after its virtual memory has increased.

The first observation is that the size of the heap allocation matches the output of mallinfo(). I have also identified the 'thread specific' data allocated using pthread_setspecific(). The problem allocations give no indication of what has caused them to be allocated.

There are a large number (hundreds) of anonymous allocations of 1024kB and 2048kB.

Prior to the memory usage 'going wild' these large allocations do not show.

Do you have a URL for documentation for mallocinfo?

In my first reply, I skipped over the fact that I don't know what mallocinfo is, because it didn't seem necessarily important.

I just did a google search and find lots of hits for mallocinfo, but none of them seem to be documentation.

In a typical implementation of malloc there is a max request size for ordinary allocations. Above that size, a different allocation method is used inside malloc (with results that should be transparent to the caller of malloc). That anonymous chunks you describe look like the result of malloc servicing requests for large chunks (but could be other things).

If those are large chunks requested from malloc, I assume they are not included in whatever you are getting from mallocinfo (or you wouldn't have asked the original question). So the next step would be figuring out why:

A) mallocinfo is only supposed to report on ordinary size chunks.

or B) mallocinfo has a bug

or C) Those chunks were allocated directly from the kernel by your application bypassing malloc.

Alternately, you could try instrumenting malloc to report the allocation and release of very large chunks.