Hello everyone. Usually good googling solves my all problems but this time it failed me.
I need help from you.

I work on a setup of rhel 6 guests running under vmware, using vmdks which live on EMC VMAX storage with flash only disks.
One of our vendors has some crazy high minimal requirements and they want us to use bonnie++ specifically to test random seeks/s

The bottom line is that more RAM servers has, then worse score for random seeks/s I am getting. Bonnie's test basically creates a file which is 2xRAM (Testing seq write at this point) and then it does a series of tests against that file, including random seeks test.
The problem is that, during initial file creation everything goes into file system cache (which of course is normal) and then the bigger that file cache is, the worse score I am getting. With memory size 16G I can easily get about 3000 seeks/s but with 64G (most used by cache at this point) this score drops down to 700 which is way bellow requirements. I've been banging my head against wall for quite some time but I can't understand why file cache is slowing things so much down.

Is it about huge page table and the fact that statistically 50% IOs have to first miss cache and then fetch data from storage (file is 2x bigger than cache so 50% makes sense I am guessing)?

If I drop cache right before random seeks test then puff, I am getting a very decent result. So it is definitely file cache related.
If I "waste" most of memory for some dummy files in SHM so leaving little room for file cache, then puff I am getting good results again.

From bonnie's source code I see that for random seek test it is generating only a total of ~8000 random IOs and messures time. So that specific test takes only few seconds. So I did another experiment. I've increased in the source code that number x100 so the test takes more time and here is what I've observed with iostat. IO Reads, within first few seconds increase from 0 to above 10.000 and then it keeps this nice number much more appropriate for flash disks.
But first few seconds on average will give those lousy ~700. That's why bonnie's scores low - this test is just to short to get to the full speed.
But then again, why would it need time to speed up? There is no memory or cpu constraint. Watching /proc/meminfo during "speed up" faze I am not observing any big changes in any of the values. Before random seek test, bonnie runs fsync call and waits enough time to write dirty pages.

Is there some sort of penalty for file cache performance under vmware?

Tested different IO schedulers, different file systems. Not much difference.

Thank you in advance for any clues on this.