Game server under attack

unSpawn · 10-23-2011, 06:11 PM

Looking forward to seeing the pcap!

smallgamer · 10-24-2011, 04:49 AM

Thanks, I have sent it to the email you asked me to.

So based on the PCAP this guy is just flooding the server with thousands of packets per second. I'm trying to limit the rate of packets one IP address can send, but it's not as straightforward as I had hoped.

I just want to make it so that the same IP address can't send me more than 100 packets per second. I found this:

http://www.debian-administration.org/articles/187

But I've learned that the "hitcount" can not be set higher than 20 (which is much too low) without increasing the default value of ip_pkt_list_tot, which is located in /etc/modprobe.conf. And, OF COURSE, this file does not exist and can not be modified on an OpenVZ vps, which is what I have.

Wtf? Then how is limiting the rate of packets from a particular IP address done on OpenVZ servers? We're just a sitting duck for DOS attacks?

smallgamer · 10-24-2011, 05:07 AM

Will --seconds take a decimal? If I can set it to 20 hits within .2 seconds, that would be the equivalent of 100 hits within 1 second, which is what I'm aiming for.

unSpawn · 10-24-2011, 07:24 PM

Quote:

Originally Posted by smallgamer

As for this attack, the tcpdump worked and I've got a log file now of the latest attack.

You didn't post the command you ran. And the capture encompasses a very short time frame. Less than half an hour is not exactly a rich data set on a well-used game server.

Quote:

Originally Posted by smallgamer

EDIT: Pcap file has been sent

Next time please contact me first to make drop-off arrangements. I don't care much for file sharing sites.

On with the pcap:

Code:

]$ capinfos ServerDoS.pcap
File type: Wireshark/tcpdump/... - libpcap
File encapsulation: Linux cooked-mode capture
Number of packets: 79662 
File size: 5170327 bytes
Data size: 3895711 bytes
Capture duration: 1404.329493 seconds
Start time: 
End time: 
Data rate: 2774.07 bytes/s
Data rate: 22192.58 bits/s
Average packet size: 48.90 bytes

]$ tcpstat -alr ServerDoS.pcap|head -3
Bytes/sec	=      2.7 KB
Bytes/minute	=    161.9 KB
Bytes/hour	=      9.5 MB

tcpstat -l -o "Since:%R\tPkt_tot=%n\tPkt_sec=%p\tPkt_avgsz=%a\tbitsec=%b\n" -r ServerDoS.pcap -1
Since:702       Pkt_tot=79662   Pkt_sec=56.73   Pkt_avgsz=48.90 bitsec=22192.58

]$ tcpstat -l -o "Since:%R\tPkt_tot=%n\tPkt_sec=%p\tPkt_avgsz=%a\tbitsec=%b\n" -r ServerDoS.pcap 60
Since:30        Pkt_tot=55237   Pkt_sec=920.62  Pkt_avgsz=47.48 bitsec=349700.27
Since:90        Pkt_tot=8073    Pkt_sec=134.55  Pkt_avgsz=51.66 bitsec=55603.07
Since:150       Pkt_tot=1458    Pkt_sec=24.30   Pkt_avgsz=52.19 bitsec=10144.80
Since:210       Pkt_tot=996     Pkt_sec=16.60   Pkt_avgsz=52.31 bitsec=6946.40
Since:270       Pkt_tot=866     Pkt_sec=14.43   Pkt_avgsz=52.34 bitsec=6044.00
Since:330       Pkt_tot=632     Pkt_sec=10.53   Pkt_avgsz=52.37 bitsec=4413.20
Since:390       Pkt_tot=662     Pkt_sec=11.03   Pkt_avgsz=52.39 bitsec=4624.40
Since:450       Pkt_tot=596     Pkt_sec=9.93    Pkt_avgsz=52.43 bitsec=4166.40
Since:510       Pkt_tot=740     Pkt_sec=12.33   Pkt_avgsz=52.37 bitsec=5167.33
Since:570       Pkt_tot=826     Pkt_sec=13.77   Pkt_avgsz=52.37 bitsec=5768.13
Since:630       Pkt_tot=912     Pkt_sec=15.20   Pkt_avgsz=52.33 bitsec=6363.73
Since:690       Pkt_tot=818     Pkt_sec=13.63   Pkt_avgsz=52.35 bitsec=5709.47
Since:750       Pkt_tot=815     Pkt_sec=13.58   Pkt_avgsz=52.33 bitsec=5686.13
Since:810       Pkt_tot=816     Pkt_sec=13.60   Pkt_avgsz=52.33 bitsec=5693.07
Since:870       Pkt_tot=769     Pkt_sec=12.82   Pkt_avgsz=52.34 bitsec=5366.93
Since:930       Pkt_tot=748     Pkt_sec=12.47   Pkt_avgsz=52.32 bitsec=5218.27
Since:990       Pkt_tot=632     Pkt_sec=10.53   Pkt_avgsz=52.44 bitsec=4418.93
Since:1050      Pkt_tot=631     Pkt_sec=10.52   Pkt_avgsz=52.41 bitsec=4409.33
Since:1110      Pkt_tot=856     Pkt_sec=14.27   Pkt_avgsz=52.32 bitsec=5971.87
Since:1170      Pkt_tot=717     Pkt_sec=11.95   Pkt_avgsz=52.39 bitsec=5008.13
Since:1230      Pkt_tot=657     Pkt_sec=10.95   Pkt_avgsz=52.37 bitsec=4587.73
Since:1290      Pkt_tot=593     Pkt_sec=9.88    Pkt_avgsz=52.40 bitsec=4143.47
Since:1350      Pkt_tot=452     Pkt_sec=7.53    Pkt_avgsz=52.15 bitsec=3142.80
Since:1410      Pkt_tot=160     Pkt_sec=2.67    Pkt_avgsz=52.98 bitsec=1130.27

]$ tcptrace -nru ServerDoS.pcap|sort -nrk 6|head -4|some sed|some awk
client.15:56765 server:36944 17744>
client.15:56765 server:36943 13000>
client.113:1053 server:36943 4155>
client.138:3033 server:36943 2730>

]$ tcpdump|count packets|grep -c|sort
2714  client.101
2789  client.138
4163  client.113
30890 client.15

Quote:

Originally Posted by smallgamer

It is clearly visible when the attacker starts flooding the server with packets

In the stats above "client.15" (last octet) has a significant larger packet count compared to the two following players. Generating statistics for the first 3 players (and unless you managed to start tcpdump at the end of an attack the first line definitely is a capture error) with

Code:

tcpstat -l -o "Pkt_tot=%n Pkt_ps=%p Pkt_avg_sz=%a bps=%b\n" -r ServerDoS.pcap 60 -f filtername

we get :

Code:

# client.15:
Pkt_tot=30748  Pkt_ps=512.47  Pkt_avg_sz=44.00  bps=180392.53
Pkt_tot=32     Pkt_ps=0.53    Pkt_avg_sz=52.38  bps=223.47
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=18     Pkt_ps=0.30    Pkt_avg_sz=52.22  bps=125.33
Pkt_tot=29     Pkt_ps=0.48    Pkt_avg_sz=52.17  bps=201.73
Pkt_tot=16     Pkt_ps=0.27    Pkt_avg_sz=52.25  bps=111.47
Pkt_tot=14     Pkt_ps=0.23    Pkt_avg_sz=52.71  bps=98.40
Pkt_tot=3      Pkt_ps=0.05    Pkt_avg_sz=51.67  bps=20.67
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=18     Pkt_ps=0.30    Pkt_avg_sz=52.22  bps=125.33
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0      Pkt_ps=0.00    Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=12     Pkt_ps=0.20    Pkt_avg_sz=52.67  bps=84.27

# client.113:
Pkt_tot=2698  Pkt_ps=44.97  Pkt_avg_sz=51.50  bps=18525.87
Pkt_tot=1465  Pkt_ps=24.42  Pkt_avg_sz=51.68  bps=10095.20

# client.138:
Pkt_tot=2504  Pkt_ps=41.73  Pkt_avg_sz=52.36  bps=17482.00
Pkt_tot=251   Pkt_ps=4.18   Pkt_avg_sz=52.92  bps=1771.20
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=15    Pkt_ps=0.25   Pkt_avg_sz=52.07  bps=104.13
Pkt_tot=9     Pkt_ps=0.15   Pkt_avg_sz=52.33  bps=62.80
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=0     Pkt_ps=0.00   Pkt_avg_sz=0.00   bps=0.00
Pkt_tot=10    Pkt_ps=0.17   Pkt_avg_sz=52.40  bps=69.87

Quote:

Originally Posted by smallgamer

So based on the PCAP this guy is just flooding the server with thousands of packets per second.

So discarding the first line it doesn't seem like client.15 exceeds any thresholds. Looking at data using Snort doesn't trigger anything. Wireshark doesn't show any interesting payload, except using a filter of

Code:

udp.length == 8

or in tcpdump

Code:

'udp && len<=46'

does show a sh*tload of payload-less packets. I wouldn't call it flooding, more like a set of bursts, as other players manage to get packets in as well. Now I haven't got any in-game experience wrt lag and players response times don't seem optimal anyway (we're talking the Unreliable Data Protocol) so I can't see how this would influence the game. And again, twenty minutes ain't exactly the amount of data to draw any conclusion from.

Quote:

Originally Posted by smallgamer

But I've learned that the "hitcount" can not be set higher than 20 (which is much too low) without increasing the default value of ip_pkt_list_tot, which is located in /etc/modprobe.conf. And, OF COURSE, this file does not exist and can not be modified on an OpenVZ vps, which is what I have. Wtf? Then how is limiting the rate of packets from a particular IP address done on OpenVZ servers?

modprobe.conf sets module loading parameters. If you can't load modules yourself then you can't change ip_pkt_list_tot yourself. Ask your provider. And please don't b*tch about OpenVZ here: nobody here forced you to rent one.

Quote:

Originally Posted by smallgamer

I'm trying to limit the rate of packets one IP address can send

Should you still want to block these packets, and given your past experiences with iptables modules, I suggest an alternative and that's using fail2ban plus Snort. You'll have to do the legwork yourself (install fail2ban and Snort, add your UDP ports to /etc/services, configure ports in /etc/fail2ban/jail.conf and an appropriate action in /etc/fail2ban/action.d/ for a rule searching for "CS_pkt_sz_lt_1", test the rule, set server IP address in snort_cs.conf) but as promised here's a Snort rule. You don't need any snort rule sets except this config and run snort with the (suboptimal: "-A fast") "-c /path/to/snort_cs.conf -A fast -q -K none -l /var/log" args:

Code:

var HOME_NET INSERTIPADDRESSHERE/32
var EXTERNAL_NET !$HOME_NET
preprocessor frag3_global: max_frags 65536
preprocessor frag3_engine: policy first detect_anomalies
preprocessor stream5_global: max_tcp 8192, track_tcp yes, track_udp no
preprocessor stream5_tcp: policy first, use_static_footprint_sizes
preprocessor sfportscan: proto  { all } memcap { 10000000 } sense_level { low }
portvar CS_PORTS [1200,27000:27050,28000,29000,36900:37000]
alert udp $EXTERNAL_NET any -> $HOME_NET $CS_PORTS (msg:"CS_pkt_sz_lt_1"; dsize: <1; reference:url,www.linuxquestions.org/questions/linux-security-4/game-server-under-attack-908100/; sid: 999999999; rev:1;)

[EDIT]
I. Alternatively you can also use a third party app like Guardian instead of fail2ban.
II. If you choose to block using Netfilter you could use a rule something like

Code:

-A INPUT -p udp -m length --length 0:8 -j DROP

though you've got to find out if --length needs the IP packet size (28) or the UDP size (8).
[/EDIT]

That's it for now.

smallgamer · 10-25-2011, 04:07 AM

Quote:

You didn't post the command you ran. And the capture encompasses a very short time frame. Less than half an hour is not exactly a rich data set on a well-used game server.

It appeared to be long enough to see the attack and plenty of standard game packets surrounding it. Perhaps you would like to tell me the minimum time you need, information I was not given before.

Quote:

Next time please contact me first to make drop-off arrangements. I don't care much for file sharing sites.

I already complied with your request and sent it to your email, so I'm not sure what your point is here.

Quote:

And please don't b*tch about OpenVZ here: nobody here forced you to rent one.

I apologize profusely - I didn't realize my logical questions regarding OpenVZ's apparent dead end in this area would be so offensive to you.

I didn't come here to be berated, and while I appreciate your help, I would also appreciate it if you would show me the same common courtesy that I show you. I assume that we're both adults here and that's not too much to ask.

Quote:

So discarding the first line it doesn't seem like client.15 exceeds any thresholds.

I'm not sure I understand. My attacker, Client.15, sends 512 packets per second, over ten times as many as the others. Is that not the threshold in question?

Quote:

does show a sh*tload of payload-less packets. I wouldn't call it flooding, more like a set of bursts, as other players manage to get packets in as well. Now I haven't got any in-game experience wrt lag and players response times don't seem optimal anyway (we're talking the Unreliable Data Protocol) so I can't see how this would influence the game. And again, twenty minutes ain't exactly the amount of data to draw any conclusion from.

...

Should you still want to block these packets

Do you mean blocking these packets from other players? No, I'm not concerned about that. I want to block the udp packets that exceed 100/second from any one IP address, i.e., the ones sent by Client.15. Do those instructions for snort and fail2ban refer to blocking the packets from other players, or the DOS attack from Client.15?

unSpawn · 10-25-2011, 03:08 PM

Quote:

Originally Posted by smallgamer

I'm not sure I understand. My attacker, Client.15, sends 512 packets per second, over ten times as many as the others. Is that not the threshold in question?

I said ditch the first reported line. And it's not 512 p/s, it's not even a sustained rate but, like I said before, bursts.

Quote:

Originally Posted by smallgamer

Do those instructions for snort and fail2ban refer to blocking the packets from other players, or the DOS attack from Client.15?

The Snort rule was tested against the contents of the full capture and only client.15 tripped it.

Quote:

Originally Posted by smallgamer

I want to block the udp packets that exceed 100/second from any one IP address, i.e., the ones sent by Client.15.

See 'man iptables' the "hashlimit" or "limit" module.

smallgamer · 10-25-2011, 03:22 PM

Thanks for the additional info. I'm sorry if my attitude was poor. I had just finished an exchange with this creep's ISP. They are STILL refusing to terminate his internet connection after nine reported attacks with logs (the previous form of attack showed up in the game server logs) over the course of the last eight months.

smallgamer · 10-26-2011, 06:30 AM

Quote:

I said ditch the first reported line. And it's not 512 p/s, it's not even a sustained rate but, like I said before, bursts.

I'm confused by this part. When I look at the PCAP file, I actually see thousands of packets per second coming from this IP. Isn't that what I need to stop?

Also, will these rules...

Code:

-A INPUT -p udp -m state --state NEW -j ACCEPT
-A INPUT -p udp -m limit --limit 100/sec -j ACCEPT
-A INPUT -p udp -j DROP

... limit udp packets to no more than 100 per second from the same IP address, or will it limit udp packets to no more than 100 per second TOTAL from all IP addresses combined?

unSpawn · 10-26-2011, 10:06 AM

Quote:

Originally Posted by smallgamer

I'm confused by this part. When I look at the PCAP file, I actually see thousands of packets per second coming from this IP. Isn't that what I need to stop?

All commands used above are available from base or third party repos. So repeating analysis isn't hard. It is suggested you download 'tcpstat' and run it to confirm my conclusion is the right one or to refute it.

Quote:

Originally Posted by smallgamer

... limit udp packets to no more than 100 per second from the same IP address, or will it limit udp packets to no more than 100 per second TOTAL from all IP addresses combined?

All IP addresses:
rule 0. UDP traffic that is not part of an already established connection leaves the INPUT chain as accepted,
rule 1. all remaining UDP traffic (implied: --state ESTABLISHED) is limited to a maximum of 100 packets per second (regardless of destination port or source IP address) and then leaves the INPUT chain as accepted,
rule 2. any remaining UDP traffic then leaves the INPUT chain being dropped.

* Rules that have the biggest impact on performance should be placed at the start of the chain as much as possible (think --state ESTABLISHED). Rules that impact performance should use as narrow a filter as possible. Right now you'll also be limiting say DNS responses while CS2D runs on just a few destination ports. Source addresses that are not (repeat) offenders should not be "punished" for the antics of others.

smallgamer · 10-26-2011, 10:53 AM

Quote:

All commands used above are available from base or third party repos. So repeating analysis isn't hard. It is suggested you download 'tcpstat' and run it to confirm my conclusion is the right one or to refute it.

I'm sure that I would get the same results if I entered the same command you did. I don't doubt that. What I don't understand is why the PCAP file shows thousands of incoming packets per second if the attacker isn't really sending thousands per second, or how stopping that flood of packets through rate limiting could be irrelevant to stopping his attack. I'm really not getting it yet, lol. Sorry if I'm being dense...

Quote:

All IP addresses:
rule 0. UDP traffic that is not part of an already established connection leaves the INPUT chain as accepted,
rule 1. all remaining UDP traffic (implied: --state ESTABLISHED) is limited to a maximum of 100 packets per second (regardless of destination port or source IP address) and then leaves the INPUT chain as accepted,
rule 2. any remaining UDP traffic then leaves the INPUT chain being dropped.

* Rules that have the biggest impact on performance should be placed at the start of the chain as much as possible (think --state ESTABLISHED). Rules that impact performance should use as narrow a filter as possible. Right now you'll also be limiting say DNS responses while CS2D runs on just a few destination ports. Source addresses that are not (repeat) offenders should not be "punished" for the antics of others.

Ok, I understand this part. So basically those rules won't do anything except make my server reject just about everybody's packets once a flood from an attacker puts me over 100 total packets per second. I can see how that's not a solution.

It seems like the only way I've found so far to rate limit packets on a per-IP-address basis is this:

Code:

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent \
  --set

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent \
  --update --seconds 60 --hitcount 4 -j DROP

But that brings me back to the issue of not being able to set "hitcount" higher than 20 or "seconds" lower than 1, which is a problem since what I need to limit is IP addresses that send more than 75-100 packets per second, not IP addresses that send more than 20 packets per second (which is almost all of them in this game).

unSpawn · 10-27-2011, 07:09 AM

This is what I would make of it:

Code:

# Limit logging then drop invalid packets:
-A INPUT -i eth0 -m udp -p udp -m state --state INVALID -m limit --limit 1/minute --limit-burst 3 -j LOG --log-prefix "UDP_IN_inv "
-A INPUT -i eth0 -m udp -p udp -m state --state INVALID -j DROP
#
# For HL/CS ports use a separate chain and filter ALL traffic regardless of state:
-A INPUT -i eth0 -m udp -p udp -m multiport --dports 36900:36999 -j UDP_IN
#
# First limit logging and drop 0-length packets:
-A UDP_IN -m length --length 0:8 -m limit --limit 1/minute --limit-burst 3 -j LOG --log-prefix "UDP_IN_len "
-A UDP_IN -m length --length 0:8 -j DROP
#
# Allow 1000 pps and bursts up to 1500 pps (per source IP and destination port combo):
-A UDP_IN -m hashlimit --hashlimit 1000/s --hashlimit-burst 1500 --hashlimit-mode srcip,dstport --hashlimit-name CS2D -j LOG --log-prefix "UDP_IN_lim "
-A UDP_IN -m hashlimit --hashlimit 1000/s --hashlimit-burst 1500 --hashlimit-mode srcip,dstport --hashlimit-name CS2D -j ACCEPT
#
# Reject everything else:
-A UDP_IN -j REJECT
#
# Meanwhile back in the INPUT chain...
-A INPUT -i eth0 -m udp -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
# etc, etc.

YMMV(VM).

Quote:

Originally Posted by smallgamer

But that brings me back to the issue of not being able to set "hitcount"

Ask your provider to change settings in the host node and make them available to your container.

smallgamer · 10-28-2011, 03:16 PM

Thanks for that, especially the hashlimit mode srcip... I don't know how that never came up in all the Googling I did.