Server very high load when wordpress was attacked

Habitual · 10-13-2015, 10:31 AM

If the manual test didn't puke, it's a good config.

Possible that the /usr/local/apache/logs/access_log file got rotated by logrotate?
look for a /usr/local/apache/logs/access_log.1

if it exists, test it manually on that.

Let me know...

Edit: Try your manaul scan on this attachment.txt.
I got these results:

Code:

Addresses found:
[1]
    192.241.235.57 (Tue Oct 06 07:33:47 2015)

Date template hits:
106 hit(s): Day/MONTH/Year:Hour:Minute:Second

Success, the total number of match is 1

Which I don't understand.
35 occurrences of abdullkarem, but 106 'hits'.

Habitual · 10-13-2015, 01:22 PM

Add this to myfilter.conf

Code:

^<HOST> .* "GET .*abdullkarem.*" .*$

and the result here is:

Code:

Success, the total number of match is 35

Sorry about that.

Original post updated.

Reference:
https://hoopercharles.wordpress.com/...linux-selinux/

fastdns · 10-14-2015, 01:07 AM

Hi Habitual,

Thank you so much. I was able to do a dry run and it looks pretty good at task :

Code:

Date template hits:
3673576 hit(s): Day/MONTH/Year:Hour:Minute:Second

Success, the total number of match is 338

But now the problem is that my fail2ban is not starting :

Code:

root@lsn5 [~]# service fail2ban start
Starting fail2ban:                                         [FAILED]
root@lsn5 [~]#

Note that after installing fail2ban, I did not do any edits to /etc/fail2ban/jail.local except that I added [myfilter] code in between (I tried it at the end of the file too, but fail2ban refuses to start)

Here is a copy of my jail.local :

Code:

# Fail2Ban jail specifications file
#
# Comments: use '#' for comment lines and ';' for inline comments
#
# Changes:  in most of the cases you should not modify this
#           file, but provide customizations in jail.local file, e.g.:
#
# [DEFAULT]
# bantime = 3600
#
# [ssh-iptables]
# enabled = true
#

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

# "backend" specifies the backend used to get files modification.
# Available options are "pyinotify", "gamin", "polling" and "auto".
# This option can be overridden in each jail as well.
#
# pyinotify: requires pyinotify (a file alteration monitor) to be installed.
#              If pyinotify is not installed, Fail2ban will use auto.
# gamin:     requires Gamin (a file alteration monitor) to be installed.
#              If Gamin is not installed, Fail2ban will use auto.
# polling:   uses a polling algorithm which does not require external libraries.
# auto:      will try to use the following backends, in order:
#              pyinotify, gamin, polling.
backend = auto

# "usedns" specifies if jails should trust hostnames in logs,
#   warn when reverse DNS lookups are performed, or ignore all hostnames in logs
#
# yes:   if a hostname is encountered, a reverse DNS lookup will be performed.
# warn:  if a hostname is encountered, a reverse DNS lookup will be performed,
#        but it will be logged as a warning.
# no:    if a hostname is encountered, will not be used for banning,
#        but it will be logged as info.
usedns = warn


# This jail corresponds to the standard configuration in Fail2ban 0.6.
# The mail-whois action send a notification e-mail with a whois request
# in the body.

[ssh-iptables]

enabled  = false
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=you@example.com, sender=fail2ban@example.com]
logpath  = /var/log/sshd.log
maxretry = 5

[proftpd-iptables]

enabled  = false
filter   = proftpd
action   = iptables[name=ProFTPD, port=ftp, protocol=tcp]
           sendmail-whois[name=ProFTPD, dest=you@example.com]
logpath  = /var/log/proftpd/proftpd.log
maxretry = 6

# This jail forces the backend to "polling".

[sasl-iptables]

enabled  = false
filter   = sasl
backend  = polling
action   = iptables[name=sasl, port=smtp, protocol=tcp]
           sendmail-whois[name=sasl, dest=you@example.com]
logpath  = /var/log/mail.log

# Here we use TCP-Wrappers instead of Netfilter/Iptables. "ignoreregex" is
# used to avoid banning the user "myuser".

[ssh-tcpwrapper]

enabled     = false
filter      = sshd
action      = hostsdeny
              sendmail-whois[name=SSH, dest=you@example.com]
ignoreregex = for myuser from
logpath     = /var/log/sshd.log

# This jail demonstrates the use of wildcards in "logpath".
# Moreover, it is possible to give other files on a new line.

[apache-tcpwrapper]

enabled  = false
filter   = apache-auth
action   = hostsdeny
logpath  = /var/log/apache*/*error.log
           /home/www/myhomepage/error.log
maxretry = 6

# The hosts.deny path can be defined with the "file" argument if it is
# not in /etc.

[postfix-tcpwrapper]

enabled  = false
filter   = postfix
action   = hostsdeny[file=/not/a/standard/path/hosts.deny]
           sendmail[name=Postfix, dest=you@example.com]
logpath  = /var/log/postfix.log
bantime  = 300

# Do not ban anybody. Just report information about the remote host.
# A notification is sent at most every 600 seconds (bantime).

[vsftpd-notification]

enabled  = false
filter   = vsftpd
action   = sendmail-whois[name=VSFTPD, dest=you@example.com]
logpath  = /var/log/vsftpd.log
maxretry = 5
bantime  = 1800

# Same as above but with banning the IP address.

[vsftpd-iptables]

enabled  = false
filter   = vsftpd
action   = iptables[name=VSFTPD, port=ftp, protocol=tcp]
           sendmail-whois[name=VSFTPD, dest=you@example.com]
logpath  = /var/log/vsftpd.log
maxretry = 5
bantime  = 1800

# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.

[apache-badbots]

enabled  = false
filter   = apache-badbots
action   = iptables-multiport[name=BadBots, port="http,https"]
           sendmail-buffered[name=BadBots, lines=5, dest=you@example.com]
logpath  = /var/www/*/logs/access_log
bantime  = 172800
maxretry = 1

# Use shorewall instead of iptables.

[apache-shorewall]

enabled  = false
filter   = apache-noscript
action   = shorewall
           sendmail[name=Postfix, dest=you@example.com]
logpath  = /var/log/apache2/error_log

# Ban attackers that try to use PHP's URL-fopen() functionality
# through GET/POST variables. - Experimental, with more than a year
# of usage in production environments.

[php-url-fopen]

enabled = false
port    = http,https
filter  = php-url-fopen
logpath = /var/www/*/logs/access_log
maxretry = 1

# A simple PHP-fastcgi jail which works with lighttpd.
# If you run a lighttpd server, then you probably will
# find these kinds of messages in your error_log:
# ALERT âil addresses. The mail outputs are buffered.
# through GET variables (attacker '1.2.3.4', file '/var/www/default/htdocs/index.php')
# This jail would block the IP 1.2.3.4.          ¦GLOBALSâ
                                                          ¦
[lighttpd-fastcgi]

enabled = false
port    = http,https
filter  = lighttpd-fastcgi
# adapt the following two items as needed
logpath = /var/log/lighttpd/error.log
maxretry = 2

# Same as above for mod_auth
# It catches wrong authentifications

[lighttpd-auth]

enabled = false
port    = http,https
filter  = lighttpd-auth
# adapt the following two items as needed
logpath = /var/log/lighttpd/error.log
maxretry = 2

# This jail uses ipfw, the standard firewall on FreeBSD. The "ignoreip"
# option is overridden in this jail. Moreover, the action "mail-whois" defines
# the variable "name" which contains a comma using "". The characters '' are
# valid too.

[ssh-ipfw]

enabled  = false
filter   = sshd
action   = ipfw[localhost=192.168.0.1]
           sendmail-whois[name="SSH,IPFW", dest=you@example.com]
logpath  = /var/log/auth.log
ignoreip = 168.192.0.1

# These jails block attacks against named (bind9). By default, logging is off
# with bind9 installation. You will need something like this:
#
# logging {
#     channel security_file {
#         file "/var/log/named/security.log" versions 3 size 30m;
#         severity dynamic;
#         print-time yes;
#     };
#     category security {
#         security_file;
#     };
# };
#
# in your named.conf to provide proper logging.
# This jail blocks UDP traffic for DNS requests.

# !!! WARNING !!!
#   Since UDP is connection-less protocol, spoofing of IP and imitation
#   of illegal actions is way too simple.  Thus enabling of this filter
#   might provide an easy way for implementing a DoS against a chosen
#   victim. See
#    http://nion.modprobe.de/blog/archives/690-fail2ban-+-dns-fail.html
#   Please DO NOT USE this jail unless you know what you are doing.
#
# [named-refused-udp]
#
# enabled  = false
# filter   = named-refused
# action   = iptables-multiport[name=Named, port="domain,953", protocol=udp]
#            sendmail-whois[name=Named, dest=you@example.com]
# logpath  = /var/log/named/security.log
# ignoreip = 168.192.0.1

# This jail blocks TCP traffic for DNS requests.

[named-refused-tcp]

enabled  = false
filter   = named-refused
action   = iptables-multiport[name=Named, port="domain,953", protocol=tcp]
           sendmail-whois[name=Named, dest=you@example.com]
logpath  = /var/log/named/security.log
ignoreip = 168.192.0.1

# Multiple jails, 1 per protocol, are necessary ATM:
# see https://github.com/fail2ban/fail2ban/issues/37
[asterisk-tcp]

enabled  = false
filter   = asterisk
action   = iptables-multiport[name=asterisk-tcp, port="5060,5061", protocol=tcp]
           sendmail-whois[name=Asterisk, dest=you@example.com, sender=fail2ban@example.com]
logpath  = /var/log/asterisk/messages
maxretry = 10

[asterisk-udp]

enabled  = false
filter   = asterisk
action   = iptables-multiport[name=asterisk-udp, port="5060,5061", protocol=udp]
           sendmail-whois[name=Asterisk, dest=you@example.com, sender=fail2ban@example.com]
logpath  = /var/log/asterisk/messages
maxretry = 10

[myfilter]

enabled  = true
filter   = myfilter.conf
port     = http
action   = myaction[name=myfilter, port="http", protocol=tcp]
logpath  = /usr/local/apache/logs/access_log
backend  = polling
findtime = 600
bantime  = 31556926 ; 1 year in seconds
maxretry = 1
ignoreip = 127.0.0.1/8 14.97.56.193/32


# Jail for more extended banning of persistent abusers
# !!! WARNING !!!
#   Make sure that your loglevel specified in fail2ban.conf/.local
#   is not at DEBUG level -- which might then cause fail2ban to fall into
#   an infinite loop constantly feeding itself with non-informative lines
[recidive]

enabled  = false
filter   = recidive
logpath  = /var/log/fail2ban.log
action   = iptables-allports[name=recidive]
           sendmail-whois-lines[name=recidive, logpath=/var/log/fail2ban.log]
bantime  = 604800  ; 1 week
findtime = 86400   ; 1 day
maxretry = 5

If I remove [myfilter], then fail2ban starts correctly.

Is there something wrong with my conf file ?

Thanks again for all that you are doing, I am sure this guide will help a lot of people.

fastdns · 10-14-2015, 07:07 AM

Hi Habitual,

Also, my CSF firewall is not running now. It refuses to start :

Code:

You have an unresolved error when starting csf:
Error: FASTSTART: (SMTP Block IPv6) [] [ip6tables-restore: line 20 failed]. Try restarting csf with FASTSTART disabled, at line 4767 in /usr/sbin/csf

You need to restart csf successfully to remove this warning, or delete /etc/csf/csf.error

Does it have anything to do with

Code:

/sbin/iptables-save > /root/safe.rules

code that I executed ?

fastdns · 10-14-2015, 07:15 AM

And my iptables comes up as "unrecognised service" :

Code:

root@lsn5 [/sbin]# service iptables start
iptables: unrecognized service
root@lsn5 [/sbin]#

And there are no ips in my rules :

Code:

root@lsn5 [/sbin]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
cphulk     all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain cphulk (1 references)
target     prot opt source               destination
root@lsn5 [/sbin]#

Please help :-( Not sure what went wrong.

Habitual · 10-14-2015, 07:36 AM

Sorry, another typo:

Code:

filter   = myfilter

As for
/sbin/iptables-save > /root/safe.rules
breaking anything, I have my doubts.

What is in /etc/csf/csf.error?
The "filter = myfilter" correction may permit fail2ban to start and possibly correct
CSF, but I have to plead ignorance on lfd/CSF with fail2ban.

fastdns · 10-14-2015, 08:17 AM

Hi Habitual,

Thanks, fail2ban finally started. But my csf firewall is not quite right. Everytime I restart, it flushes all the firewall rules.

And this abdullkarem has started attacking again :

Code:

   21-38 19477 0/2/2329   W 0.03  21   0    0.0  0.00  14.41  70.32.88.133   xxxxx.com       GET /404.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1
   22-38 19478 0/1/2515   W 0.05  15   0    0.0  0.00  17.85  70.32.88.133   xxxxx.com       GET /8.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&w
   23-38 19480 0/1/1938   W 0.05  15   0    0.0  0.00  11.28  70.32.88.133   xxxxx.com       GET /7.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&w
   24-38 19482 0/1/3159   W 0.08  14   0    0.0  0.03  16.74  70.32.88.133   xxxxx.com       GET /9.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&w
   25-38 19483 0/2/1631   W 0.05  11   0    0.0  0.01  9.47   70.32.88.133  xxxxx.com       GET /shell.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem

How do I verify that his IP 70.32.88.133 was blocked by fail2ban ? His connections are still active in apache :-(

Thanks for all the help that you are providing, much appreciated.

fastdns · 10-14-2015, 12:55 PM

Hi Habitual,

This freaking "abdullkarem" attack continues and I don't think any IP is getting blocked.

I checked several attacking IPs using :

Code:

iptables -L INPUT -v -n | grep "70.32.88.133"

and it shows no block rule.

But fail2ban service is on :

Code:

root@lsn5 [~]# ps aux |grep fail
root      9290  0.0  0.0 124132  3904 ?        S    08:58   0:00 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
root     19493  0.0  0.0  61224   772 pts/0    R+   13:46   0:00 grep fail

Did we miss any piece of code in the myaction.conf file ?

And I think I can remove CSF , since fail2ban has several of those features too. But I need to ensure that fail2ban is working correctly first.

Any advise you can provide will be really helpful.

Thanks again.

Sefyir · 10-14-2015, 01:15 PM

Quote:

Originally Posted by fastdns

Hi Habitual,

This freaking "abdullkarem" attack continues...

Did you attempt the rules I posted?

A fail2ban service may add rules explicitly banning certain ips based on strings, but using the rule I posted will prevent any certain ip from overwhelming your server with requests

fastdns · 10-14-2015, 02:07 PM

Hi Sefyir,

Quote:

Originally Posted by Sefyir

Did you attempt the rules I posted?

A fail2ban service may add rules explicitly banning certain ips based on strings, but using the rule I posted will prevent any certain ip from overwhelming your server with requests

Yes, I did. Still no luck. My server is on a load of 90 and unusable now :-(. These abusers are ruining my little business.

astrogeek · 10-14-2015, 02:41 PM

I see a lot of recommendations for fail2ban, but have not had much notable success with it on my own VPSs.

I think the reason is that my adaptive iptables rules catch all the non-http intrusion attempts, and the http attempts require more specific rules than I was able to understand... and an intuitive understanding of what fail2ban was suposed to be doing at any given time eluded me - so I never had a high confidence factor in depending on it. It still runs on my VPSs, but it rarely catches anything of interest.

Just my own experience, not intended to be as negative about fail2ban as it sounds. It targets specific kinds of attacks - intrusions - and is not so applicable to others.

But looking at your specific abdulkarem attack - if that really is the source of your problem - you would probably do well to install and learn your way around modsecurity. It too, has a learning curve, so buy the book and invest some hours in understanding it - but it can give very precise control over blocking and logging of such traffic.

In my case I had to cope with a very targeted attack on a specific site, in addition to the script kiddies, and an understanding of modsecurity was the key to doing that for the http traffic. It took some learning time, but in the end I understood it and am now able to monitor and adapt my rules with high confidence - and it is actually the confidence that you need more than any particular application.

Sefyir · 10-14-2015, 03:04 PM

Quote:

Originally Posted by fastdns

Hi Sefyir,
Yes, I did. Still no luck. My server is on a load of 90 and unusable now :-(. These abusers are ruining my little business.

That's surprising. If a single unique ip address is spamming your server, being limited to roughly one request every 2 seconds should not be enough to raise serverload so high!
I ran a while loop with curl pulling a php page as fast as it could to my server and bumped my cpu from 0.3 to %7.5.. then after the 120 buffer, tanked back to %0.3.

I agree with astrogeek though - I've had limited success with fail2ban and in my experience is better suited for failed logins. After all this work, you're mostly only going to be protected from a single attack.
I would also consider learning netfilter (iptables) rather then using automated programs (like ufw) since it can render fine-tuned controls in a single line.
For example, in my provided rule, a ip address is permitted to make 120 requests that are refilled at 28 requests a minute. If it goes over that limit, it drops the packet.
I've discovered that it's quite hard to hit that ceiling unless I do something unfriendly (like hold down Shift+F5)

Once you have a hard ceiling (as such with a firewall), then you should move down to application level and do more fine-tuning of requests by the webserver, which increases server load but gives more control and less issues when going over a level.

Also, you may want to fine-tune your webserver. It really shouldn't be getting to 90% and limiting access to others. You may have a config issue somewhere causing high-cpu when traffic increases. If your small business suddenly booms, you really don't want that to be the moment you discover you can't handle increased traffic!

Personally, I have a 512MB VPS that after I switched from apache to nginx + php5-fpm, could handle significantly larger amounts of traffic when before it was almost maxing out my memory with little usage.

fastdns · 10-14-2015, 03:54 PM

Quote:

Originally Posted by Sefyir

That's surprising. If a single unique ip address is spamming your server, being limited to roughly one request every 2 seconds should not be enough to raise serverload so high!
I ran a while loop with curl pulling a php page as fast as it could to my server and bumped my cpu from 0.3 to 7.5.. then after the 120 buffer, tanked back to %0.3.

Thanks for your thoughts Sefyir. Could there be something wrong with my iptables itself ? I flushed everything and restarted iptables, but it shows everything as 0.0.0.0 :

Code:

root@lsn5 [/csf]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    cP-Firewall-1-INPUT  all  --  0.0.0.0/0            0.0.0.0/0
2    acctboth   all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    cP-Firewall-1-INPUT  all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    acctboth   all  --  0.0.0.0/0            0.0.0.0/0

Chain RH-Firewall-1-INPUT (0 references)
num  target     prot opt source               destination

Chain acctboth (2 references)
num  target     prot opt source               destination

Chain cP-Firewall-1-INPUT (2 references)
num  target     prot opt source               destination
1    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:993
2    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2078
3    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:53
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:21
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2082
6    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443
7    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2077
8    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
9    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:26
10   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:8080
11   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:143
12   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
13   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:995
14   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:110
15   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:25
16   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2086
17   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2087
18   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2095
19   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:465
20   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2096
21   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:3306
22   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2083
23   ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           state NEW udp dpt:53

Table: nat
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Quote:

I agree with astrogeek though - I've had limited success with fail2ban and in my experience is better suited for failed logins. After all this work, you're mostly only going to be protected from a single attack.

Problem is, it is not even doing that. Again, this may be related to my iptables not working correctly as indicated above ?

Quote:

I would also consider learning netfilter (iptables) rather then using automated programs (like ufw) since it can render fine-tuned controls in a single line.
For example, in my provided rule, a ip address is permitted to make 120 requests that are refilled at 28 requests a minute. If it goes over that limit, it drops the packet.
I've discovered that it's quite hard to hit that ceiling unless I do something unfriendly (like hold down Shift+F5)

For my sake, could you please verify the iptables rule that you provided ? I have to run each of those two rules separately, right ?

Quote:

Also, you may want to fine-tune your webserver. It really shouldn't be getting to 90% and limiting access to others. You may have a config issue somewhere causing high-cpu when traffic increases. If your small business suddenly booms, you really don't want that to be the moment you discover you can't handle increased traffic!

I have already done whatever that i have learned till now. The server runs Comodo's modsecurity rules, Nginx , Mysql is fine tuned, but I may have still left out some loop hole somewhere. I will look for finding some professional help from someone (do you know of anyone who fine tunes servers ?)

Thanks a lot for all the help, much appreciated.

Sefyir · 10-14-2015, 04:24 PM

I see there's some gui involvement here that making the iptable ruleset bloated..
could you post output of iptables-save?

Quote:

1 cP-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain cP-Firewall-1-INPUT (2 references)
6 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
8 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80

My impression is this is causing all http(s) traffic to be accepted prior to encountering any other rules.
You should stop this "cP-Firewall" from running. Apologies, I don't know a lot of gui firewalls. But if you just flushed it and it seems to have recovered, it's probably running. I also don't see my rule anywhere o.O

As a example, here is my (modified) iptables script

Code:

ipt=/sbin/iptables

$ipt -F # Flush rules
$ipt -X # Delete tables

# Policies and Chains
$ipt -P INPUT DROP # set input policy to drop
$ipt -P FORWARD DROP # set forward policy to drop (change if needed)
$ipt -P OUTPUT ACCEPT
$ipt -N WEBSERVER # Separate chain for webserver

$ipt -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Accept all established, related connections
$ipt -A INPUT -i lo -j ACCEPT # Allow loopback # Accept all connections done within the server - computer talking to itself

# Services
$ipt -A INPUT -p tcp -m multiport --dport 443,80 -m conntrack --ctstate NEW -j WEBSERVER # Jump to WEBSERVER chain
$ipt -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT # Accept ssh connections

# WEBSERVER chain
$ipt -A WEBSERVER -p tcp -m hashlimit --hashlimit-name NORMAL_USAGE --hashlimit-mode srcip --hashlimit 25/minute --hashlimit-burst 120 -j ACCEPT
$ipt -A WEBSERVER -p tcp -m hashlimit --hashlimit-name OVER_USAGE_OF_WEBSERVER --hashlimit-mode srcip --hashlimit 12/minute --hashlimit-burst 1 -j LOG --log-prefix "OVER_USAGE_OF_WEBSERVER "

astrogeek · 10-14-2015, 05:03 PM

Quote:

Originally Posted by fastdns

Also, I do have some basic tools in place such as :
- ConfigServer Security Firewall
- ConfigServer Exploit Scanner
- Comodo's WAF Mod Security Ruleset
- wp-login protection script (that challenges wordpress's login page)

but none of them seem to help in this case.

Please help.

Honestly, the problem is that you really do not know what you have security-wise.

Do you know what the ConfigServer Security Firewall rules are actually doing? Have you tested it?
Do you know what the ConfigServer Exploit Scanner is doing? Have you tested it?
Do you know what the Comodo's WAF Mod Security Ruleset is actually doing? Have you tested it?

This is not finger pointing, if the answer to any of these questions is no, then you really do not know what you have security-wise!

It is not sufficient to just install any of these types of "solutions" and expect them to take care of all the problems - you have to understand the environment and configure the tools to address your specific problems.

I tried to go look at the Comodo rules but they require a signup and login, so... but there is NO one-size-fits-all rule set - you cannot plug-n-play with web server security, you are going to have to understand the problem and the ruleset and use them selectively. It is also very likely that the default Comodo rules log only, you have to configure them to actually block traffic. This is common for modsecurity tools as it prevents you from accidentally blocking all your traffic with a naive installation! So the Comodo WAF rules might actually be doing nothing at this time - unless you have understood them and enabled them.

It is also entirely possible (aka likely) that your problem could be amplified by one of the "solutions". For example, I think I see reverse DNS lookups in your log data if I am reading it correctly, and I know many WordPress plugins and features perform reverse lookups for such things as spam comment sources. This generates additional network traffic and CPU usage for every incoming request... an amplifier.

While learning modsecurity, for example, I found an OWASP default rule that made a remote request of its own to a comment spam database for the purpose of logging. The way I found it was that an attacker was aware of the rule and knew how to trigger it and was causing my server to spam itself! The lesson here was that you must understand the rules and selectively enable them!

And as for the iptables Firewall - it is not possible to tell what the chained rule are from the data posted. And it would not be reasonable to have you post them all as it would waste a lot of other people's time and in the end, you still would not know what you had...

In security applications - more is definitely not better! Fewer, well understood applications and rules will win every time!

Start with your iptables rules... if you do not understand them then give yourself a crash course until you can write your own simple rules. In my opinion you should avoid all GUI or browser managed firewalls - they only write incomprehensible rules and add confusion. If you don't understand iptables then you cannot understand what the managed firewall is doing! If you understand iptables (not really difficult) then you don't need the managed firewall application anyway!

All comments intended to be helpful, but intended to focus on the path to resolving the problem, not merely masking it in a convenient manner.