Application running too slow on Sun SPARC T5440 but run normal on sun M3000

kecodoc · 06-13-2012, 02:01 AM

Hi all,
I have application running on sun server T5440 4x8x1.4 GHz, 64 GB RAM, application running very slow though load average too low. when I install my application on another server SUN M3000 (One CPU 1x8x2.5GHz, 8GB RAM), application run smoothly.

Here is my server T5440 info:

Code:

bash-3.00# uname -a
    SunOS 5.10 Generic_142909-17 sun4v sparc SUNW,T5440

Psrinfo -pv

Code:

bash-3.00# psrinfo -pv
    The physical processor has 64 virtual processors (0-63)
    UltraSPARC-T2+ (chipid 0, clock 1414 MHz)
    The physical processor has 64 virtual processors (64-127)
    UltraSPARC-T2+ (chipid 1, clock 1414 MHz)
    The physical processor has 64 virtual processors (128-191)
    UltraSPARC-T2+ (chipid 2, clock 1414 MHz)
    The physical processor has 64 virtual processors (192-255)
    UltraSPARC-T2+ (chipid 3, clock 1414 MHz)

/etc/system

Code:

set semsys:seminfo_semopm=10000
    set semsys:seminfo_semmsl=10000
    set semsys:seminfo_semmni=1024
    set max_nprocs=50000
    set msgsys:msginfo_msgmnb=2097152
    set msgsys:msginfo_msgmni=2560
    set msgsys:msginfo_msgtql=2560
    set shmsys:shminfo_shmmax=1073741824
    set shmsys:shminfo_shmmni=1000
    set rlim_fd_cur=10000

Ipcs

Code:

bash-3.00# ipcs -a
    IPC status from as of Wed Jun 13 12:31:01 ICT 2012
    T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME
    Message Queues:
    q 62 0x1388 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:22
    q 61 0x1387 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:22
    q 60 0x1318 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:22
    q 59 0x12a9 --rw-rw-rw- topx topx topx topx 0 0 2097152 9069 9071 10:25:38 10:25:38 8:36:22
    q 58 0x100f --rw-rw-rw- topx topx topx topx 0 0 2097152 9069 9070 10:06:16 10:06:16 8:36:22
    q 57 0xfa0 -Rrw-rw-rw- topx topx topx topx 0 0 2097152 9071 9069 10:25:38 10:25:38 8:36:22
    q 56 0xf30 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:20
    q 55 0xc27 -Rrw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:28:14
    q 54 0xbb8 -Rrw-rw-rw- topx topx topx topx 0 0 2097152 172 9066 10:47:08 10:47:08 8:28:14
    T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
    Shared Memory:
    T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
    Semaphores:
    s 104 0x9c4 --ra-ra-ra- topx topx topx topx 3 12:31:00 8:36:20
    s 103 0xa8c --ra-ra-ra- topx topx topx topx 1 8:36:20 8:36:20
    s 102 0xd48 --ra-ra-ra- topx topx topx topx 1 8:36:20 8:36:20
    s 101 0xce4 --ra-ra-ra- topx topx topx topx 1 8:36:20 8:36:20
    s 100 0x26de --ra-ra-ra- topx topx topx topx 8 10:19:34 8:28:14
    s 99 0xc1c --ra-ra-ra- topx topx topx topx 1 8:36:20 8:28:14
    s 98 0xbb8 --ra-ra-ra- topx topx topx topx 1 8:43:50 8:28:14
    s 97 0xaf0 --ra-ra-ra- topx topx topx topx 500 8:36:20 8:28:14
    s 96 0xa28 --ra-ra-ra- topx topx topx topx 500 8:36:20 8:28:14
    s 95 0x7ee --ra-ra-ra- topx topx topx topx 1 8:36:20 8:28:14
    s 94 0x7e4 --ra-ra-ra- topx topx topx topx 1001 8:36:20 8:28:14
    s 93 0x7da --ra-ra-ra- topx topx topx topx 220 8:36:20 8:28:14
    s 92 0x7d5 --ra-ra-ra- topx topx topx topx 200 12:05:15 8:28:14
    s 91 0x7d0 --ra-ra-ra- topx topx topx topx 200 8:56:42 8:28:14
    s 90 0xc80 --ra-ra-ra- topx topx topx topx 2 12:31:01 8:27:52 –a

Prstat

Code:

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

    9202 topx 1460M 893M sleep 59 0 0:16:37 0.2% integrity_check/1
    12455 root 3880K 3600K cpu65 59 0 0:00:00 0.0% prstat/1
    9080 topx 142M 130M sleep 59 0 0:00:52 0.0% intf_resp_low/1
    9125 topx 102M 78M sleep 59 0 0:00:36 0.0% interface_vsc/1
    9121 topx 88M 80M sleep 59 0 0:00:32 0.0% interface_tcu/1
    9120 topx 89M 80M sleep 59 0 0:00:29 0.0% interface_rcm/1
    9124 topx 87M 78M sleep 59 0 0:00:29 0.0% baplie_merge_v2/1
    9122 topx 86M 78M sleep 59 0 0:00:28 0.0% interface_eih/1
    9115 topx 88M 77M sleep 59 0 0:00:33 0.0% intf_batch_even/1
    9114 topx 88M 77M sleep 59 0 0:00:39 0.0% intf_event/1
    9117 topx 87M 78M sleep 59 0 0:00:32 0.0% interface_bxy/1
    9118 topx 86M 78M sleep 59 0 0:00:29 0.0% interface_ppm/1
    NPROC USERNAME SWAP RSS MEMORY TIME CPU

    41 topx 334M 2237M 3.4% 0:30:17 0.3%
    43 root 186M 195M 0.3% 0:14:45 0.0%
    1 noaccess 152M 136M 0.2% 0:04:00 0.0%
    1 smmsp 3688K 11M 0.0% 0:00:04 0.0%
    1 lp 1208K 3880K 0.0% 0:00:00 0.0%
    6 daemon 7264K 9944K 0.0% 0:00:26 0.0%
    Total: 93 processes, 259 lwps, load averages: 0.40, 0.39, 0.39

Prstat –mL

Code:

bash-3.00# prstat -mL
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
    9202 topx 0.9 0.0 0.0 0.0 0.0 0.0 99 0.0 2 0 5 0 integrity_ch/1
    9080 topx 0.4 0.0 0.0 0.0 0.0 0.0 100 0.0 5 0 20 0 intf_resp_lo/1
    9114 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 13 0 66 0 intf_event/1
    17282 root 0.1 0.2 0.0 0.0 0.0 0.0 100 0.0 20 1 324 0 prstat/1
    9117 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 1 70 0 interface_bx/1
    9118 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 1 70 0 interface_pp/1
    9115 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 15 0 80 0 intf_batch_e/1
    9121 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_tc/1
    9125 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_vs/1
    9120 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_rc/1
    9123 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_vs/1
    9124 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 1 70 0 baplie_merge/1
    9122 topx 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 8 0 56 0 interface_ei/1
    9119 topx 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 8 0 56 0 interface_cp/1
    9204 topx 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 15 1 1K 0 sys_check/1
    259 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 13 2 98 7 in.mpathd/1
    1 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 99 3 init/1
    1337 noaccess 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 100 0 100 0 java/14
    140 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 19 0 114 0 nscd/851
    Total: 90 processes, 256 lwps, load averages: 0.40, 0.40, 0.39

Vmstat

Code:

bash-3.00# vmstat 1 10
    kthr memory page disk faults cpu
    r b w swap free re mf pi po fr de sr m0 m1 m2 m3 in sy cs us sy id
    0 0 0 87720712 60562856 173 645 199 44 44 0 0 0 0 0 0 886 2819 525 0 1 99
    1 0 0 87563976 59743040 73 327 0 15 15 0 0 0 0 0 0 747 2831 439 0 1 99
    0 0 0 87563656 59742808 61 306 0 15 15 0 0 0 0 0 0 770 2684 427 0 1 99
    1 0 0 87563656 59742824 0 0 0 0 0 0 0 0 0 0 0 886 1302 525 0 1 99
    1 0 0 87563656 59742816 62 305 0 15 15 0 0 0 0 0 0 722 2652 420 3 1 96
    1 0 0 87563656 59742840 554 2777 0 139 139 0 0 0 0 0 0 959 9736 749 1 1 97
    0 0 0 87563456 59742672 1155 5735 0 292 292 0 0 0 0 0 0 1113 17489 1145 1 3 96
    1 0 0 87563656 59742840 0 0 0 0 0 0 0 0 0 0 0 845 1613 654 0 1 99
    0 0 0 87563656 59742856 62 305 0 15 15 0 0 0 0 0 0 900 2647 526 0 1 99
    0 0 0 87563656 59742856 62 304 0 15 15 0 0 0 0 0 0 683 2661 396 0 1 99

Mpstat

Code:

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
    0 0 0 130 261 1 121 0 13 1 0 0 0 1 0 99
    1 0 0 35 133 2 143 0 9 6 0 64 0 0 0 100
    2 0 0 5 20 2 17 0 3 0 0 29 1 0 0 99
    3 0 0 0 3 0 2 0 0 0 0 0 0 0 0 100
    4 0 0 3 14 0 15 0 2 2 0 13 0 0 0 100
    73 0 0 474330 77 3 0 7 0 2 0 2596 67 33 0 0
    73 0 0 487701 37 0 0 7 0 8 0 2670 66 34 0 0
    73 0 0 476377 14 0 0 8 0 1 0 2610 66 34 0 0
    78 0 0 473950 11 1 0 7 0 13 0 2593 67 33 0 0
    78 0 0 480562 9 0 0 7 0 4 0 2630 66 34 0 0
    79 0 0 320989 14 1 0 8 0 0 0 1766 77 23 0 0
    79 0 0 479148 9 0 0 7 0 8 0 2622 66 34 0 0
    79 0 0 479426 13 0 0 7 0 2 0 2622 67 33 0 0
    79 0 0 488949 9 0 0 8 0 13 0 2678 66 34 0 0
    64 0 0 482229 15 2 0 8 0 7 0 2638 65 35 0 0
    77 0 0 475571 14 1 0 7 0 8 0 2602 67 33 0 0
    69 0 0 467795 22 1 0 7 0 40 0 2563 66 34 0 0
    69 0 0 318623 9 1 0 7 0 8 0 1751 77 23 0 0
    66 0 0 465179 92 1 2 8 0 62 0 2545 66 34 0 0

When my application running, some of CPU have IDLE state = 0 and hold the CPU while another CPU have IDLE state 100. I think this is my bottleneck but I cannot find any solution for solve that.

Netstat

Code:

bash-3.00# netstat -i
    Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
    lo0 8232 loopback localhost 244022 0 244022 0 0 0
    nxge0 1500 TOPX-01-DUMMY1 TOPX-01-DUMMY1 3594863 0 5801650 0 0 0
    nxge1 1500 TOPX-01-DUMMY2 TOPX-01-DUMMY2 160954 0 151230 0 0 0

iostat

Code:

bash-3.00# iostat -xn
    extended device statistics

    r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
    0.1 0.1 1.5 30.8 0.0 0.0 1.8 9.0 0 0 md/d0
    0.0 0.1 0.8 30.8 0.0 0.0 0.0 9.9 0 0 md/d1
    0.0 0.1 0.8 30.8 0.0 0.0 0.0 9.8 0 0 md/d2
    0.0 0.0 0.0 0.0 0.0 0.0 0.0 15.2 0 0 md/d3
    0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.9 0 0 md/d4
    0.0 0.0 0.0 0.0 0.0 0.0 0.0 19.4 0 0 md/d5
    0.0 0.0 2.2 1.5 0.0 0.0 0.8 7.4 0 0 md/d6
    0.0 0.0 1.1 0.7 0.0 0.0 0.0 8.5 0 0 md/d7
    0.0 0.0 1.1 0.8 0.0 0.0 0.0 8.2 0 0 md/d8
    0.0 0.1 0.4 0.6 0.0 0.0 7.6 10.0 0 0 md/d9
    0.0 0.1 0.2 0.6 0.0 0.0 0.0 9.1 0 0 md/d10
    0.0 0.1 0.2 0.6 0.0 0.0 0.0 7.8 0 0 md/d11
    0.0 0.0 0.0 0.0 0.0 0.0 0.1 10.4 0 0 md/d12
    0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.5 0 0 md/d13
    0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.4 0 0 md/d14
    0.1 0.5 2.1 32.3 0.0 0.0 0.0 9.9 0 0 c0t0d0
    0.1 0.5 2.1 32.3 0.0 0.0 0.0 9.8 0 0 c0t1d0
    0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
    0.9 0.9 125.3 123.1 0.0 0.1 0.0 34.2 0 2 c4t600144F0CC00C29A00004D1E456F0003d0
    0.3 0.7 20.7 53.9 0.0 0.0 0.2 8.2 0 0 c4t600144F0CC00C29A00004D1E43880001d0
    0.1 0.2 50.7 111.5 0.0 0.0 0.0 59.1 0 1 c4t600144F0CC00C29A00004D1E453E0002d0

Please help me find the bottleneck of my T5440 server?Thanks so much for all your help.

jlliagre · 06-13-2012, 04:42 AM

I would use dtrace to figure out what cause the significant number of system calls bound on some CPUs. Is your system using ldoms ? Is it up to date with patches ? Are there processor sets configured ?

kecodoc · 06-13-2012, 05:13 AM

Thanks jlliagre, some information about lockstat, dtrace in our T5440 system http://pastebin.com/AdUgcVgJ.
And M3000: http://pastebin.com/HPSgHaP8.
Our system don't using ldoms and don't have processor sets configured.
Thanks for your help.