vmstat and sar -r free memory discrepancy

eggixyz · 01-07-2009, 10:20 PM

Cool - look forward to it.

I got that number wrong and then noticed you "did" do a pagesize for 8192, so I emptied my post. Looks like I did it too late

Best wishes to you,

Mike

AlucardZero · 01-09-2009, 12:49 PM

Code:

alucard@v1:~$ date
Fri Jan  9 13:48:40 EST 2009
alucard@v1:~$ pagesize
8192
alucard@v1:~$ vmstat -S 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 0 0 0 20142816 4682856 0 0 993 12 12 0  0 134 83 75 3  753 8426 1320  2  3 95
 0 0 0 19983664 4525160 0 0 79  8  0  0  0 102 92 96 1  830 8134 1094  1  2 97
^C
alucard@v1:~$ date
Fri Jan  9 13:49:28 EST 2009
alucard@v1:~$ sudo /usr/lib/sa/sa1
Password:
alucard@v1:~$ sar -r | tail
12:40:00  138539  9971000
12:50:00  140113  9995619
13:00:00  140344  9981022
13:10:00  140076  9986206
13:20:00  139499  9964207
13:30:01  138555 10001133
13:40:00  139819  9985552
13:49:32  141275  9988549

Average   140661  9932497
alucard@v1:~$ echo "141275*8" | bc
1130200

eggixyz · 01-09-2009, 10:55 PM

Yep,

The numbers don't agree. The next things that come to mind that might, possibly, help are, in Solaris 10:

1. vmstat lists the size of the free list in K

2. sar -r shows amount of free pages available

3. sar can be installed locally or globally (not sure, of the top of head about vmstat, but I believe it can also be tied to a zone)

Are you getting this information while logged into a local zone or the global zone (or are you using zones at all?)

Check the output of "prstat -m" or "prstat -a" and "prstat -z"

The only rationale that I can come up with (not "knowing" the answer yet) is that one of the numbers (vmstat possibly) is recording numbers globally and sar is recording locally within a zone.

Otherwise, it appears that you have 3gb of memory unaccounted for. The first (well, next) I would do, if I had the chance and none of the above applied, would be reboot the machine to make sure that vmstat isn't reporting incorrectly. My experience is that, once vmstat gets goofed (by wraparound or losing track during thrashing) it stays that way until its counters get cleared.

I'll stay on this list. I'm very interested to see how this turns out.

Thanks for posting the numbers, too. Definitely shows that there's a "major" difference that doesn't make sense assuming that all things are equal.

Best wishes,

Mike

AlucardZero · 01-09-2009, 10:58 PM

Thanks.

This machine is not using zones, and it's a Very Important Server I can't reboot at whim. However it has been rebooted since I started this thread and to my knowledge has never had its numbers agree.

I can get prstat output come Monday.

eggixyz · 01-09-2009, 11:09 PM

Cool,

I'll definitely check it out. Also, if it's not data that you can't share, can you post output of "uname -a" (solaris kernel/patch version) and, if possible, "prtdiag -v" (or just the model of your server, actual memory, etc from that output), "psrinfo" and "kstat."

Since this problem is so seemingly difficult it probably has a really easy explanation

In any event, the more info you can share, the better. Obviously, if you'd be getting in trouble revealing any of that information, paraphrasing it would be better than nothing.

Talk to you soon,

Mike

AlucardZero · 01-12-2009, 10:52 AM

The output of kstat is 870KB - I don't think I can post it all here

Code:

alucard@v1:~$ uname -a
SunOS v1 5.10 Generic_127127-11 sun4u sparc SUNW,Sun-Fire-V445
alucard@v1:~$ sudo /usr/sbin/psrinfo
0       on-line   since 01/07/2009 13:04:56
1       on-line   since 01/07/2009 13:04:56
2       on-line   since 01/07/2009 13:04:56
3       on-line   since 01/07/2009 13:04:54

prtdiag -v:

Code:

System Configuration: Sun Microsystems  sun4u Sun Fire V445                                         
System clock frequency: 199 MHZ                                                                     
Memory size: 8GB                                                                                    
                                                                                                    
==================================== CPUs ====================================                      
               E$          CPU                    CPU                                               
CPU  Freq      Size        Implementation         Mask    Status      Location                      
---  --------  ----------  ---------------------  -----   ------      --------                      
0    1592 MHz  1MB         SUNW,UltraSPARC-IIIi    3.4    on-line     MB/C0/P0                      
1    1592 MHz  1MB         SUNW,UltraSPARC-IIIi    3.4    on-line     MB/C1/P0                      
2    1592 MHz  1MB         SUNW,UltraSPARC-IIIi    3.4    on-line     MB/C2/P0                      
3    1592 MHz  1MB         SUNW,UltraSPARC-IIIi    3.4    on-line     MB/C3/P0 

============================ Memory Configuration ============================                      
Segment Table:                                                                                      
-----------------------------------------------------------------------                             
Base Address       Size       Interleave Factor  Contains                                           
-----------------------------------------------------------------------                             
0x0                2GB               4           BankIDs 0,1,2,3                                    
0x1000000000       2GB               4           BankIDs 16,17,18,19                                
0x2000000000       2GB               4           BankIDs 32,33,34,35                                
0x3000000000       2GB               4           BankIDs 48,49,50,51                                
                                                                                                    
Bank Table:                                                                                         
-----------------------------------------------------------                                         
           Physical Location                                                                        
ID       ControllerID  GroupID   Size       Interleave Way                                          
-----------------------------------------------------------                                         
0        0             0         512MB           0,1,2,3                                            
1        0             1         512MB                                                              
2        0             1         512MB                                                              
3        0             0         512MB                                                              
16       1             0         512MB           0,1,2,3                                            
17       1             1         512MB                                                              
18       1             1         512MB                                                              
19       1             0         512MB                                                              
32       2             0         512MB           0,1,2,3                                            
33       2             1         512MB                                                              
34       2             1         512MB                                                              
35       2             0         512MB                                                              
48       3             0         512MB           0,1,2,3                                            
49       3             1         512MB                                                              
50       3             1         512MB                                                              
51       3             0         512MB                                                              
                                                                                                    
Memory Module Groups:                                                                               
--------------------------------------------------                                                  
ControllerID   GroupID  Labels         Status                                                       
--------------------------------------------------                                                  
0              0        MB/C0/P0/B0/D0 okay                                                         
0              0        MB/C0/P0/B0/D1 okay                                                         
0              1        MB/C0/P0/B1/D0 okay                                                         
0              1        MB/C0/P0/B1/D1 okay                                                         
1              0        MB/C1/P0/B0/D0 okay                                                         
1              0        MB/C1/P0/B0/D1 okay                                                         
1              1        MB/C1/P0/B1/D0 okay                                                         
1              1        MB/C1/P0/B1/D1 okay                                                         
2              0        MB/C2/P0/B0/D0 okay                                                         
2              0        MB/C2/P0/B0/D1 okay                                                         
2              1        MB/C2/P0/B1/D0 okay                                                         
2              1        MB/C2/P0/B1/D1 okay                                                         
3              0        MB/C3/P0/B0/D0 okay                                                         
3              0        MB/C3/P0/B0/D1 okay                                                         
3              1        MB/C3/P0/B1/D0 okay                                                         
3              1        MB/C3/P0/B1/D1 okay                                                         
                                                            


=========================== FRU Operational Status ===========================                      
---------------------------------                                                                   
Fru Operational Status:                                                                             
---------------------------------                                                                   
Location                Status                                                                      
---------------------------------                                                                   
MB/SC                   okay                                                                        
MB/HDDCNTRL/HDDBP/HDD0  present                                                                     
MB/HDDCNTRL/HDDBP/HDD1  present                                                                     
MB/HDDCNTRL/HDDBP/HDD2  present                                                                     
MB/HDDCNTRL/HDDBP/HDD3  present                                                                     
MB/HDDCNTRL/HDDBP/HDD4  present                                                                     
MB/HDDCNTRL/HDDBP/HDD5  present                                                                     
MB/HDDCNTRL/HDDBP/HDD6  present                                                                     
MB/HDDCNTRL/HDDBP/HDD7  present                                                                     
PS0                     okay                                                                        
PS1                     okay                                                                        
PS2                     okay                                                                        
PS3                     okay    

System PROM revisions:                                                                              
----------------------                                                                              
OBP 4.22.33 2007/06/18 12:42 Sun Fire V445                                                          
POST 4.22.33 2007/06/18 13:10

And for kicks:

Code:

alucard@v1:~$ /usr/sbin/zoneadm list
global

jlliagre · 01-12-2009, 11:12 AM

Can you post the "kstat ::system_pages" output ?

Both vmstat and sar pick their information from there.

jlliagre · 01-12-2009, 11:12 AM

Can you post the "kstat ::system_pages" output ?

Both vmstat and sar pick their information from there.

AlucardZero · 01-12-2009, 02:00 PM

Code:

alucard@v1:~$ sudo kstat ::system_pages
Password:
module: unix                            instance: 0     
name:   system_pages                    class:    pages
        availrmem                       898883
        crtime                          233.7688366
        desfree                         8015
        desscan                         25
        econtig                         52428800
        fastscan                        132066
        freemem                         368515
        kernelbase                      16777216
        lotsfree                        16030
        minfree                         4007
        nalloc                          27316585
        nalloc_calls                    12325
        nfree                           26014201
        nfree_calls                     7600
        nscan                           0
        pagesfree                       368515
        pageslocked                     127095
        pagestotal                      1025978
        physmem                         1044992
        pp_kernel                       144297
        slowscan                        100
        snaptime                        439222.737739

alucard@v1:~$ date
Mon Jan 12 15:01:06 EST 2009
alucard@v1:~$ vmstat -S 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 0 0 0 19779872 4348248 0 0 1017 11 11 0 0 120 77 70 2  695 8060 1278  2  3 95
 0 0 0 18398176 2947040 0 0 1942 78 78 0 0 124 69 71 6 2279 49829 6214 21 14 65
^C
alucard@v1:~$ sudo /usr/lib/sa/sa1
alucard@v1:~$ sar -r | tail -3
15:01:21   92072  9165695

Average   103977  9319605

jlliagre · 01-13-2009, 04:31 AM

You do not need special privileges (sudo/pfexec) to run kstat.

In any case, kstat output is consistent with vmstat:

368515*pagesize=2948120 free kB out of 8 GB of RAM

There is something wrong with sar on your system. Can you run "sar -r 2 4" as I already asked ?

AlucardZero · 01-13-2009, 08:55 AM

Thanks, didn't know, can't hurt though eh?

Code:

alucard@v1:~$ sar -r 2 4

SunOS v1 5.10 Generic_127127-11 sun4u    01/13/2009

09:57:08 freemem freeswap
09:57:10   85102  8947137
09:57:12   85009  8936114
09:57:14   85132  8947133
09:57:16   84883  8914158

Average    85031  8936115
alucard@v1:~$ vmstat -S 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 0 0 0 19547904 4141544 0 0 1104 11 11 0 0 125 82 74 2  727 24159 1365 2  3 94
 0 2 0 18140760 2766712 0 0 1008 0 0  0  0 274 150 126 0 1233 19582 1313 2 3 95
 0 2 0 18140760 2767280 0 0 1180 0 0  0  0 341 219 124 0 1698 31839 1893 2 5 93
 0 2 0 18140760 2767944 0 0 1109 0 0  0  0 268 138 128 0 1143 16799 1187 1 3 96
^C

jlliagre · 01-13-2009, 10:50 AM

Sar seems to think your pagesize is 32 MB.

What displays this command ?

Code:

apptrace -v sysconf sar -r 1 1

AlucardZero · 01-13-2009, 12:16 PM

Code:

alucard@v1:~$ apptrace -v sysconf sar -r 1 1
-> sar      -> libc.so.1:long sysconf(int = 0xb)
        arg0 = (int) 0xb
        return = (long) 0x2000
<- sar      -> libc.so.1:sysconf() = 0x2000
-> sar      -> libc.so.1:long sysconf(int = 0x3)
        arg0 = (int) 0x3
        return = (long) 0x64
<- sar      -> libc.so.1:sysconf() = 0x64

SunOS asw-vob1 5.10 Generic_127127-11 sun4u    01/13/2009

13:18:33 freemem freeswap
13:18:34   86898  8916874

eggixyz · 01-13-2009, 08:10 PM

Go away for a day or two and things get nuts

That's interesting that your pagesize is being interpreted as 32Mb.

I wasn't aware that this was possible (and haven't finished reading this whole thing yet), but apparently Solaris 10 supports multiple page sizes

http://www.solarisinternals.com/wiki...e_Size_Support

Perhaps this might answer the question (?)

Can't wait to see how this one turns out

Best wishes,

Mike

eggixyz · 01-13-2009, 08:14 PM

Hey Again,

From that article:

Can you run:

pmap -sx SomeProcessPID

for some of the higher volume apps you're running? This should show you exact pagesize's per process. It seems like, from the simple admin-end, that it makes things a bit more difficult to eyeball, but it could be that only certain processes are using higher-than-default pagesize.

, Mike