Hello, I'm having problems on my server where it has freezes in the service responsible for the MUMPS server on our IBM/AIX. I have noticed that the MUMPS service freezes when the server load average increases. At certain times of the day the load average increases to values ​​above 30. At the moment it is at 14.81.
SISMED1:/var/tmp> uptime
03:09PM up 34 days, 22:57, 50 users, load average: 14.81, 12.31, 13.20

When it increases a lot, the service responsible for MUMPS/MSM freezes and we have to kill the service to restart the MUMPS system. Is there anything I can check or is there any system tuning to improve processor performance?

Here are some data

SISMED1:/var/tmp> oslevel -r
7100-05
SISMED1:/var/tmp> uname -a
AIX SISMED1 1 7 00FB42D74C00

SISMED1:/var/tmp> prtconf | grep -i "Processor Type"
Processor Type: PowerPC_POWER8
SISMED1:/var/tmp> lsattr -El sys0 | grep -i model
modelname IBM,8286-41A Machine name False
SISMED1:/var/tmp> lsattr -El proc0|grep -E "freq|type"
frequency 3724000000 Processor Speed ​​False
type PowerPC_POWER8 Processor type False

SISMED1:/var/tmp> uptime
03:11PM up 34 days, 23 hrs, 49 users, load average: 21.55, 15.95, 14.47
SISMED1:/var/tmp> ps -ef | egrep -v "STIME|$LOGNAME" | sort +3 -r | head -n 20
root 2294088 21692518 58 10:04:01 - 86:09 mumsm 1
root 60162070 21692518 4 10:03:59 - 0:09 muserver 132
root 25690180 21692518 3 10:04:01 - 0:23 muserver 1008
root 15925388 21692518 3 10:04:01 - 0:14 muserver 773
root 15007768 21692518 3 10:04:01 - 0:13 muserver 824
root 26345658 21692518 3 10:04:01 - 0:12 muserver 775
root 18678218 21692518 3 10:04:01 - 0:08 muserver 782
root 32178256 21692518 3 10:04:00 - 0:31 muserver 388
root 16908700 21692518 3 10:04:00 - 0:17 muserver 395
root 28049756 21692518 3 10:04:00 - 0:03 muserver 356
root 34013198 21692518 3 10:03:59 - 8:09 muserver 134
root 6160786 21692518 3 10:03:59 - 0:08 muserver 142
root 25886922 21692518 2 10:04:01 - 0:01 muserver 835
root 39649530 21692518 2 10:04:00 - 0:10 muserver 583
root 1245528 21692518 2 10:04:00 - 0:08 muserver 660
root 34078832 21692518 2 10:04:00 - 0:06 muserver 330
root 26608036 21692518 2 10:04:00 - 0:06 muserver 304
root 28901636 21692518 2 10:03:59 - 0:06 muserver 60
root 14876948 21692518 1 10:04:01 - 0:13 muserver 893
root 29950292 21692518 1 10:04:01 - 0:09 muserver 846

SISMED1:/var/tmp> lparstat 5 2

System configuration: type=Shared mode=Uncapped smt=Off lcpu=2 mem=16384MB psize=8 ent=2.00

%user %sys %wait %idle physc %entc lbusy vcsw phint
----- ----- ------ ------ ----- ----- ------ ----- -----
27.8 26.3 0.4 45.4 1.15 57.4 58.5 4881 0
28.0 29.9 0.0 42.1 1.22 61.0 61.7 4620 0
SISMED1:/var/tmp> uptime
03:14PM up 34 days, 23:02, 46 users, load average: 42.96, 23.78, 17.47


SISMED1:/var/tmp> uptime
03:18PM up 34 days, 23:07, 47 users, load average: 85.81, 61.00, 35.38
SISMED1:/var/tmp> ps -ef | egrep -v "STIME|$LOGNAME" | sort +3 -r | head -n 20
root 2294088 21692518 60 10:04:01 - 88:35 mumsm 1
root 25690180 21692518 1 10:04:01 - 0:25 muserver 1008
root 721204 21692518 1 10:04:01 - 0:18 muserver 800
root 22348084 21692518 1 10:04:01 - 0:16 muserver 874
root 16384190 21692518 1 10:04:01 - 0:09 muserver 977
root 27656196 21692518 1 10:04:01 - 0:09 muserver 887
root 13041828 21692518 1 10:04:01 - 0:07 muserver 732
root 33489084 21692518 1 10:04:01 - 0:07 muserver 700


SISMED1:/var/tmp> uptime
03:24PM up 34 days, 23:12, 50 users, load average: 94.80, 81.23, 51.82
SISMED1:/var/tmp> vmstat -Iwt 2

System configuration: lcpu=2 mem=16384MB ent=2.00

kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------------------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
38 1 0 1265158 6972 240 241 0 0 0 0 9747 393136 43404 24 32 44 0 1.19 59.6 15:24:23
122 0 0 1265758 5982 140 121 0 0 0 0 8057 325536 36254 30 26 43 0 1.20 59.8 15:24:25
55 0 0 1265758 5431 184 209 0 0 0 0 9258 364265 42399 25 29 45 0 1.16 58.0 15:24:27
74 0 0 1265827 5028 105 156 0 0 0 0 9636 382992 43773 25 30 46 0 1.16 58.1 15:24:29
83 0 0 1265862 5647 73 97 0 0 467 476 8847 437987 39165 34 36 30 0 1.44 72.0 15:24:31
80 0 0 1265611 5503 110 103 0 0 0 0 9467 397141 43378 27 33 40 0 1.25 62.7 15:24:33
141 0 0 1265611 5263 65 86 0 0 0 0 8545 347205 38507 28 27 45 0 1.15 57.7 15:24:35
172 0 0 1265612 4966 87 71 0 0 0 0 9080 369104 41146 27 29 44 0 1.19 59.3 15:24:37
41 0 0 1265618 5138 72 50 0 0 0 0 7969 393872 36325 26 28 45 0 1.16 57.8 15:24:39
155 0 0 1265635 7455 114 18 0 0 1293 1362 9790 423290 44117 28 34 38 0 1.30 65.1 15:24:41


SISMED1:/var/tmp> lparstat -i
Node Name : SISMED1
Partition Name : SISMED1
Partition Number : 9
Type : Shared
Mode : Uncapped
Entitled Capacity : 2.00
Partition Group-ID : 32777
Shared Pool ID : 0
Online Virtual CPUs : 2
Maximum Virtual CPUs : 2
Minimum Virtual CPUs : 1
Online Memory : 16384 MB
Maximum Memory : 16384 MB
Minimum Memory : 2048 MB
Variable Capacity Weight : 192
Minimum Capacity : 0.50
Maximum Capacity : 2.00
Capacity Increment : 0.01
Maximum Physical CPUs in system : 8
Active Physical CPUs in system : 8
Active CPUs in Pool : 8
Shared Physical CPUs in system : 8
Maximum Capacity of Pool : 800
Entitled Capacity of Pool : 540
Unallocated Capacity : 0.00
Physical CPU Percentage : 100.00%
Unallocated Weight : 0
Memory Mode : Dedicated
Total I/O Memory Entitlement : -
Variable Memory Capacity Weight : -
Memory Pool ID : -
Physical Memory in the Pool : -
Hypervisor Page Size : -
Unallocated Variable Memory Capacity Weight: -
Unallocated I/O Memory entitlement : -
Memory Group ID of LPAR : -
Desired Virtual CPUs : 2
Desired Memory : 16384 MB
Desired Variable Capacity Weight : 192
Desired Capacity : 2.00
Target Memory Expansion Factor : -
Target Memory Expansion Size : -
Power Saving Mode : Disabled
Sub Processor Mode : -

From what you posted the MUMPS service seems to be the problem, not the hardware. Take a look at its logs to see what it is doing. It looks like you have 2 CPUs, so if the service is never running well perhaps you need to add more?
Perhaps there is a poorly constructed SQL query?
Memory leak in the app?
Did you talk to the developer to help troubleshoot?

The MUMPS/MSM service is a non-relational database. It's something very old, it's very difficult to find content.

One possible source of the problem could be contention for a lock file. You could check the program logic to see if interlocks are possible. Another possibility is that a particular lock is used so much that in a high volume situation the application essentially becomes single threaded.

1 members found this post helpful.

Any suggestions to check this?

1 members found this post helpful.

When the system freezes you might take a look at the times when the various lock files were created or accessed to see if one or a combination of several lock files have been locked for a long time.

You might have to go into the logic of the program. When it updates a record does it lock the entire file or does it just lock a single record in the file? When it locks on a record does it also lock on an index file?

Wow. I haven't encountered MUMPS since the mid-'90s. (I think I still have a book on the application/database stashed in a box somewhere.) Back then it was on VMS and the ODS filesystem combined with the way MUMPS dealt with data could fragment files pretty seriously. I was called to one site to investigate their performance issue, I found that there were most of individual files used by MUMPS that had upwards of 14,000 extents and the system was grinding virtually to a halt when hitting those files. Defragging the filesystem made a huge difference. In those days, the officially sanctioned method was to schedule downtime and backup/restore... ugh!. Not sure if this is something your site is suffering from but it might be a good idea to check it out.

1 members found this post helpful.

Hello, we are using MUMPS/MSM from Intersystems. It runs on a Unix/AIX, the file system is jfs2. Where can I check these individual files you mentioned?

1 members found this post helpful.

I don't know how to report this. I'll see with development if they know anything about it.