[SOLVED] ActiveMQ

czezz · 12-20-2022, 05:16 PM

OS: RHEL 7.9
ActiveMQ 5.16.1
Java Runtime: Oracle Corporation 1.8.0_311

All of the sudden ActiveMQ is crashing.
If i try to restart service it will last 10-50 seconds and crash again.

In the activemq.log I can see that it complains about "Too many open files".

Code:

...
2022-12-20 23:07:33,753 | ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.SocketException: Too many open files | org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.SocketException: Too many open files
...

Here is what I have already tried but none of it helps:

Code:

/etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 4096
root soft nproc unlimited

* hard core 0
* hard nofile 65535
* soft nofile 65535
activemq soft nproc 16384
activemq hard nproc 16384

# su - activemq
$ ulimit -Hu
16384
$ ulimit -Su
16384

Code:

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 160851
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 160851
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In /opt/activemq/conf/activemq.xml I tried to change value 1000 to 4000 and also to 6000. Doesnt help

Code:

...
          <!-- DOS protection, limit concurrent connections to 1000 and frame size
                to 100MB -->
            <transportConnector name="openwire"
                                uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" />
            <transportConnector name="amqp"
                                uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" />
            <transportConnector name="stomp"
                                uri="stomp://0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" />
            <transportConnector name="mqtt"
                                uri="mqtt://0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" />
            <transportConnector name="ws"
                                uri="ws://0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" />
        </transportConnectors>
...

Any ideas where else to look and to try?

czezz · 12-21-2022, 06:21 AM

Hi everyone,
In my case the issue has been solved by adding LimitNOFILE to the /usr/lib/systemd/system/activemq.service

Steps to do:
1. Modify /usr/lib/systemd/system/activemq.service

Code:

[Unit]
Description=Apache ActiveMQ
After=network-online.target

[Service]
Type=forking
WorkingDirectory=/opt/activemq/bin
ExecStart=/opt/activemq/bin/activemq-service
#ExecStart=/opt/activemq/bin/activemq start
ExecStop=/opt/activemq/bin/activemq stop
Restart=on-abort
User=activemq
Group=activemq
LimitNOFILE=50000 <<<--- Added parameter

[Install]
WantedBy=multi-user.target

2. Reload systemd manager configuration (as root):

Code:

systemctl daemon-reload

3. Restart/start activemq (as root):

Code:

systemctl restart activemq