is ipset working?

mfoley · 11-07-2022, 09:21 AM

I am running Samba as an Active Directory / Domain controller on a Linux host. I monitor the attempts to log into domain workstations for hackers trying to break in. Those log messages will look like:

Code:

[2022/11/07 06:44:06.677882, 2] authentication for user [(null)/USER1] FAILED with error NT_STATUS_NO_SUCH_USER, port: 2222, IP: 31.43.185.6

When my countermeasure script detects 10 or more failed logins, it blocks the IP as:

Code:

/usr/sbin/ipset -exist add blockrdc 31.43.0.0/16

and I set iptables as:

Code:

iptables -t raw -I PREROUTING -m set --match-set blockrdc src -j DROP

The problem is, I'm not sure this is working. The above IP was added to the ipset list on November 1st, and still shows in the list. Yet the log message shown above is from November 11th. Why would the hacker be able to try this port if ipset had added this IP to the blockrdc list?

One thought I had is that when the iptables command is run, it takes the blockrdc list as it then exists and that possibly iptables is not dynamic in that sense so that future additions to the blockrdc list are not seen until the iptables command is re-run. Is that possible?

Turbocapitalist · 11-08-2022, 08:40 PM

I'm rather sure that --match-set in iptables either does not work well or at all, I can't remember which.

That functionality is what caused me to upgrade to NFTables instead. Look at named sets in NFTables:

https://wiki.nftables.org/wiki-nftab...ets#Named_sets

https://wiki.nftables.org/

https://wiki.nftables.org/wiki-nftab..._in_10_minutes

IPTables is basically deprecated already since little to no development seems to be happening on it and all the activity seems to be with NFTables instead. The parts of IPTables which work still work, the parts that are unfinished will remain so.

mfoley · 11-09-2022, 12:29 PM

I'll check out NFTables and get back, but whoa! I have some pretty complex iptables rules that experts (mainly LQ) have been helping me set up over the years. Could iptables and NFTables be run concurrently? If so, that might help me migrate over time, rule by rule.

astrogeek · 11-09-2022, 04:24 PM

I am stll a fan of iptables and also have quite a lot of things that depend on it (and a lot invested in learning it that I am reluctant to abandon).

While I recall having encountered a few quirks with ipset and always verify my rules work before deploying them, I have not found any cases in my own use that simply "didn't work".

Without seeing the set definition and the context of that rule, I would suggest it may be more likely that those packets never reached the match rule, or perhaps the set definition is unsuitable for the rule.

Could you post the definition of that set and note how many members it has. I see it is in the raw/PREROUTING chain, so not much should be ahead of it, but still... I would contrive a test to verify the rule and look for other reasons that connection was logged before abandoning it (had the system been rebooted in that time frame? Do you have absolute certainty the rule was in place at that time? Etc...)

I would also suggest explicitly adding the protocol to that rule as well (from my own cryptic notes - add separate rules for TCP and UDP matches).

Turbocapitalist · 11-09-2022, 07:06 PM

Quote:

Originally Posted by mfoley

Could iptables and NFTables be run concurrently?

The way I have done it is that most old systems have stayed the way they are but new ones start out with NFtables. As the old ones reach the end of life, IPTables disappears through attrition.

A few systems outgrew IPTables and got upgraded. For those I removed IPTables and installed NFTables and then built the new rules line by line from reading the old scripts. That might not be the most efficient way nor suitable for all environments.

What are your scripts like?

mfoley · 11-13-2022, 01:17 PM

Quote:

Originally Posted by astrogeek

I am stll a fan of iptables and also have quite a lot of things that depend on it (and a lot invested in learning it that I am reluctant to abandon).

Amen! I've use iptables on all my Linux hosts going back at least 10-15 years. The bash script that initializes on startup it is 250 lines, and other scripts add to iptables rules throughout the month. I'd rather not attempt to redo that. Turbocapitalist's suggestion to try NFTables on a new system might be an idea for the future.

Quote:

Could you post the definition of that set and note how many members it has.

Definitiion: ipset -exist create blockrdc hash:net hashsize 1024 counters

Code:

> ipset list
Name: blockrdc
Type: hash:net
Revision: 4
Header: family inet hashsize 1024 maxelem 65536 counters
Size in memory: 4320
References: 3
Members: 31 (currently)
185.153.0.0/16 packets 0 bytes 0
185.156.0.0/16 packets 183256 bytes 9285194
62.233.0.0/16 packets 111 bytes 5672
88.214.0.0/16 packets 7992 bytes 415536
45.187.0.0/16 packets 179416 bytes 9329624

(I've listed 5 of the 31 members above just to show some example entries)

Quote:

I see it is in the raw/PREROUTING chain, so not much should be ahead of it,

Should I possible add other chains? For example:

Code:

    iptables -I INPUT -m set --match-set blockrdc src -j DROP
    iptables -I FORWARD -m set --match-set blockrdc src -j DROP

I just had another incident today:

Code:

This is the attempted break-in message received today:

[2022/11/13 05:56:32.308294, 2] authentication for user [SOMEDOM/Admin] FAILED with error NT_STATUS_NO_SUCH_USER, port: 2222, IP: 45.187.125.243

However, this same attempt was detected 6 days ago. Below is the countermeasure script log entry which 
would have process the same above break-in attempt message only with a date of 11/07:

NEW: 2022-11-07 06:46
IP: 45.187.125  Attempted Remote Desktop port accesses: 11
country:     BR

The following is the ipset rule added by the above countermeasure script. As you can see, it is dated 11/07:

/usr/sbin/ipset -exist add blockrdc 45.187.0.0/16  # 2022-11-07 06:46 11 Remote Desktop port accesses

Obviously, the ipset rule created for IP 45.187.0.0/16 on November 7th did not work as the hacker was able to try again 6 days later. This is concerning because if attackers can retry repeatedly without being blocked that gives them infinite opportunity to brute force the system.

Quote:

but still... I would contrive a test to verify the rule and look for other reasons that connection was logged before abandoning it

I'll try that, but might be a problem as I have to be a normal human and attempting the login from a Windows workstation whereas these attackers are usually bots that can make the attempt dozens of times a minute. I have no such bot-script. I'll try it and post back results.

Quote:

(had the system been rebooted in that time frame? Do you have absolute certainty the rule was in place at that time? Etc...)

Well, as you can see from my example, the 'ipset list' shows that IP in the list, and the logfile entries show it being added on Nov 7th. So that rule was definitely there on the 13th when the attempt was again made from that same IP. Even if rebooted, all the blocked IPs and ipsets are saved in a script and the the boot startup scripts will re-add the rules. In any case, no this system has not been rebooted for 43 days.

Quote:

I would also suggest explicitly adding the protocol to that rule as well (from my own cryptic notes - add separate rules for TCP and UDP matches).

So, how would I do that, just ...

Code:

iptables -t raw -p tcp -I PREROUTING -m set --match-set blockrdc src -j DROP

And are you suggesting a 2nd rule for udp?

mfoley · 11-14-2022, 01:33 PM

I tried logging in remotely deliberately using a bad password until the target Windows host blocked me. The intrusion script on the Active-Directory/Domain-Controller (Linux/Samba) then saw the failed attempts and blocked the IP:

Code:

:
[2022/11/14 13:22:32.366450, 2] authentication for user [MYDOM/me] FAILED with error NT_STATUS_WRONG_PASSWORD, port: 2222, IP: 184.57.48.4
[2022/11/14 13:22:36.180772, 2] authentication for user [MYDOM/me] FAILED with error NT_STATUS_WRONG_PASSWORD, port: 2222, IP: 184.57.48.4
[2022/11/14 13:22:40.320858, 2] authentication for user [MYDOM/me] FAILED with error NT_STATUS_WRONG_PASSWORD, port: 2222, IP: 184.57.48.4
[2022/11/14 13:22:45.082982, 2] authentication for user [MYDOM/me] FAILED with error NT_STATUS_ACCOUNT_LOCKED_OUT, port: 2222, IP: 184.57.48.4

The intrusion script logged the blockage and did ipset:

Code:

NEW: 2022-11-14 13:24
IP: 184.57.48   Attempted Remote Desktop port accesses: 14

/usr/sbin/ipset -exist add blockrdc 184.57.0.0/16  # 2022-11-14 13:24 14 Remote Desktop port accesses

# ipset list
184.57.0.0/16 packets 55 bytes 3180

I waited until after the account lockout period, then tried logging in again and my IP is still blocked, as expected. I expect it to remain blocked until I delete the ipset rule.

This all worked as expected. So then why does it sometimes not work as described in my post #6? I had another incident today with an IP allegedly blocked with ipset 7 days ago, yet 16 attempts from that same IP were made today.

I don't get why it usually works but sometimes doesn't. I'll try explicitly added the -p tcp parameter to the --match-set rule, but I'm doubtful that will have any effect.

More thoughts?

astrogeek · 11-14-2022, 03:04 PM

Quote:

Originally Posted by mfoley

So, how would I do that, just ...

Code:

iptables -t raw -p tcp -I PREROUTING -m set --match-set blockrdc src -j DROP

And are you suggesting a 2nd rule for udp?

That looks right, and yes, if you want to block udp traffic, add a udp rule too. (I would put the -p tcp after the -I term for readability, but I don't think it matters).

You have counters enabled in that set, what do they say with regard to the "missed" IPs? In other words, does the counter indicate a match?

astrogeek · 11-14-2022, 03:23 PM

Quote:

Originally Posted by mfoley

This all worked as expected. So then why does it sometimes not work as described in my post #6? I had another incident today with an IP allegedly blocked with ipset 7 days ago, yet 16 attempts from that same IP were made today.

I don't get why it usually works but sometimes doesn't. I'll try explicitly added the -p tcp parameter to the --match-set rule, but I'm doubtful that will have any effect.

More thoughts?

I don't get why it appears to not work either, but I would have to see absolute proof, a repeatable test case, that it did not work before blaming it on ipset/iptables, and I don't think we have that yet.

Are there multiple interfaces on this host so that it could be arriving via another path?
It is difficult to suggest other tests or possibilities without knowing your complete setup. But this comes to mind:

Find one or more of those actual blocked IPs which repeat fairly often and add a simple non-ipset counter rule for each immediately after the ipset match rule:

Code:

ipatbles -t raw -I PREROUTING -p tcp -s x.x.0.0/16

Then monitor the logs and the counters - if an intrusion attempt is found then check the counter after the ipset match rule and see if it came in that way. Compare with the ipset counter for that IP and if they don't match I would have to begin thinking ipset didn't work - or at least log some f those complete packets to see if something else is happening!

jayjwa · 01-26-2023, 04:28 PM

As far as I know, it should be on the INPUT table. You can have nft and iptables together on the same system. I haven't gotten around to relearning firewall tools so I still use iptables. Use like so:

Code:

#ipset create testset hash:net family inet hashsize 64 maxelem 512
#ipset add testset 74.0.0.0/8
#ipset list testset                                                                                                                                                                                
Name: testset
Type: hash:net
Revision: 7
Header: family inet hashsize 64 maxelem 512 bucketsize 12 initval 0xc44cc377
Size in memory: 504
References: 1
Number of entries: 1
Members:
74.0.0.0/8

# iptables -I INPUT -m set --match-set testset src -m limit -j LOG --log-level 7 --log-prefix "Ipset testset: "
# iptables -L INPUT -v -n --line-numbers                                                                                                                                                  
Chain INPUT (policy ACCEPT 1617 packets, 303K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        5  5633 LOG        0    --  *      *       0.0.0.0/0            0.0.0.0/0            match-set testset src limit: avg 3/hour burst 5 LOG flags 0 level 7 prefix "Ipset testset: "
...

#tail -n 1 /var/log/firewall
2023-01-26T17:08:44.927159-05:00 atr2 kernel: [702094.648757] Ipset testset: IN=eth0 OUT= MAC=6c:4b:90:17:7c:1d:00:01:5c:8a:a8:46:08:00 SRC=74.x.x.x DST=74.x.x.x LEN=1116 TOS=0x02 PREC=0x00 TTL=51 ID=48964 DF PROTO=TCP SPT=12175 DPT=58116 WINDOW=1702 RES=0x00 ACK PSH URGP=0

# iptables -D INPUT 1
# ipset destroy testset

Obviously, change the LOG target or just add a DROP or REJECT target after it to block stuff. You might want to -A INPUT (append) instead of -I (insert), depending on your other rules. Make sure any block rule comes before something else that would let it in.