Hi all,

This is kind of a two parter and it sort of falls in networking and/or software but since it primarily about networking I am asking it here.

First Part -- Can anyone suggest what could be eating up bandwidth on a WAN link?

We have an ATT leased full T1, we had some slow Internet access start approx a month ago. ATT Business direct bandwidth monitoring showed us that on May 26 our bandwidth utilization (Inbound) went to approx 95% of capacity and it stayed there 24 hours a day, 7 days a week until ATT had a Network outage on June 9th in the evening (something in their switching station figuratively, or maybe literally, blew up) As expected witht he outage All traffic ceased during that period. When Internet access was restored later that night usage stayed low until normal business hours. on the 10th of June after normal hours usage went to almost nothing. then On the 11th during working hours it went up....however, that night it stayed maxed all night long and all day the 12th, then right through the weekend. Then it started peaking during the day and going to almost nothing at night, until this past weekend it was maxed out all weekend.

Logs from our router/firewall appliance (Juniper SSG 140) have an occasional Jumbo ICMP or Fragmented traffic but nothing else out of the ordinary and these alarms are so infrequent that it is certainly not the cause. I have been monitoring all traffic in and out of our network With multiple utilities on a machine running in promiscuos mode for weeks now and the internal numbers DO NOT match external numbers. I have Run Wireshark on the weekends, during the day, at night and those logs (Some very large multi GB files) do not bear resemblance to the ATT monitoring. Short of unplugging users one by one I have done nearly everything to try to see whats happening and find a cause (We have unplugged servers with no effect and they are really the only computers left on 24/7) I know it is possible to have a bad NIC cause issues like this, however, I have not seen any trash traffic and except for where my bandwidth monitoring computer is plugged in the network is fully switched and we have no issues internally with connectivity or bandwidth usage. Additionally this issue occurs when most computers are off, like this weekend there were only 10 computers (excluding servers) that were on out of 60 and most of those run services that require them to be on. like Spiceworks, ADT card access system and a Payroll system.

Does anyone have any idea how to proceed or what could be causing this issue?

This segways(sp?) into Part two -- Logging and monitoring utilities this is kind of a two part part two.

2a)Is there any software for Linux, or even Windows if need be tha tis Opensource or freeware that will log all network conversations snatched. Kind of like Wireshark or iftop that show you a conversation but not log as much information. i.e. I don't need to see every frame. I just want the IP that initiated a connection at what time and to what IP with what ports and protocol if possible and what time that conversation terminated. I don't care so much at this juncture what goes on during the conversation. I have been unable so far to really find anything like that that will log, iftop shows me that info(minus time of conenction) but in more or less real time with no log retention and no time stamp. Everything else I have that logs that info doesn't have what I am looking for. Bandwidthd comes close but it doesn't have exact times and it doesn't tell me who the other end of the conversation is with.

2b)Are there any online utilities that are free to use that can monitor traffic for a set of given IP address so I can actually see what is coming in and out on my WAN link. ATT cannot see where that traffic is origination from or if it is to one of our static IPs instead of our serial IP address. I do NOT and cannot afford a T1 card for a linux machine to put in front of our router so that I can actually monitor all inbound traffic like I am on the WAN side, and our router does not have that kind of functionality.

Ultimately I have come to the conclusion that this inbound traffic is not making it to the LAN, our firewall is dropping it, but it is not logging it? This is network traffic that does not exist, but is causing a palpable problem. And I really need some help on this as I am close to running out of ideas. Today is unplug the employee and send them for a coffee break day and see if that has an impact, but if that yields no results I don't know where to go.


Thank you for reading this long drawn out post and for any assistance you can provide.

Alex

1) Give the netstat-nat package a try on your router/firewall. It'll show you natted connections which you otherwise wouldn't see with regular tools. Worth a try.
2a) You can create filters for tcpdump that'll do only that. Alternatively, ngrep is pretty neat as well. The argument for tcpdump to show the timestamp in the output is -tttt. man tcpdump(8) for more info.
2b) Free: not that I know off.

Having dealth with a situation like that a couple of times, I'd recommend to start looking at connections with netstat-nat/tcpdump.
Single out machines with unusually high bandwith usage and investigate further.

Hi, thank you for the response. Could you give me an idea how to setup tcpdump to show me the NATed connection, unfortunately netstat-nat is crying about not finding the netfilter kernel module and the machine I am using to run these collections services is a host OS for some production vms (tight on hardware around here) I appreciate it.

Alex

I take your main firewall/router isn't a linux box?
The reason I ask is if you can't run diagnostics on your main firewall (or whatever you want to call it) box, you won't be able to sniff all outgoing connections without pulling some tricks.

Suppose this is your network, switched:

X1 ---> | SW0 | ---> FW ---> INET
X2 ---> | |
....

Let's say X1 is the box you're after (X1 is heavily torrenting) and X2 is your box you running diagnostics on.
X2 won't be able to see X1's NAT'ed connections, you want to sniff on SW0 or FW.
You could place a box in between SW0 and FW and divert all traffic through it though or you could trick SW0 into believing you're in fact FW so that all traffic goes through you.

In case you can sniff on the correct box, however, this is the explanation on how to do it:

As tcpdump shows every packet it sees and you only want packets going 'through' your box, you could filter out everything coming from/going out of the ip address of your firewall/router like this: tcpdump -tttt -vvv -n -s 100 not host <internal_ip_firewall> [and not host <internal_ip_firewall_2> ... ] and not arp.

This'll show you everything not originating from your firewall with timestamps, snaplenght of 100 (increase if needed),no resolving of hostnames (so pure ip addresses) and no arp requests.
If you suspect tcp connections are filling your bandwidth, you can filter for only {insert_protocol} connections by adding 'and {protocol}' to that command (eg 'and tcp').

If FW is a linux box and iptables has the conntrack module loaded you can check that out as well (it has all the NAT'ed live connections listed). I believe the table is at /proc/net/ip_conntrack or something like that.

I hope all this helped a bit.

Well you are right the linux box is not the firewall (I should have realized that netfilter wasn't on the box becuase I don't need it as it is not a gateway and it doesn't do any nat-ting brain not working today I guess). However, we have one port going from our hardware firewall/router to one of the switches and from there it goes to the network. What I have done is is re-routed that cable and instead of directly plugging into the LAN port on the firewall I have plugged that into a HUB (kicking it old school!) then connected the hub to the firewall LAN port and to my spy box. I have put the NIC connected to the hub in promiscuous mode. This is not ideal obviously, however, as HUBS use broadcast everysingle piece of traffice destined from the LAN-->WAN or WAN-->LAN will go through there and be broadcast to all ports. and my box in promiscuous mode should pick it up. Let me provide a quick rundown of things I have done utilities I have used.


Thing is I myself had some bitorrent traffic on my laptop (not the computer being used for monitoring) and I could see all that traffic. The numbers that all of these utilities report to me for usage do not jive with ATT stats for our line for inbound traffic. At times they match during work hours, but not the whole day. Our avg usage for the day inbound is about 60% (per our data) ATT reports the avg usage at 90% or better, not just that but during off hours our data for inbound barely spikes every one or two hours to 1Mb (about 2/3 the T1 capacity) but ATT shows it at 90% constantly 24/7. Our numbers for outbound traffic match almost exactly what ATT reports. I just can't see this network traffic, and the logs on the firewall itself do not show any suspicious activity or any red flags on the WAN interface. The firewall also does not appear to be dropping lots of packets based on firewall policy either. Taking users off the LAN by unplugging them seems to have no effect on the ATT numbers, (if yesterday is an indication) The servers also appear not to be an issue as disconnecting them has no appreciable effect. If you have any other suggestions I would greatly appreciate them.

Thank you for all the assistance you have given me so far

Alex

At this point, I'd try contacting them and confronting them with your numbers.
It seems kinda fishy to me inbound traffic is at 90% 24/7.
It's a bit extreme, but as a test you could offline the firewall and check their usage statistics for that time, no firewall running means no inbound traffic, right?
Same logic for the switch, firewall is online but no client can connect to the WAN, so high volumes of incoming data then are being dropped by your firewall, or your firewall has a life on it's own, or ATT's numbers are incorrect.

Did you consider the possibility you're experiencing a DoS attack btw?
Maybe some host is sending large volumes of data which are being (silently) dropped by your firewall? It's kinda far fetched, but who knows...

Well here is the catch I have called ATT and we disconnected the LAN (Firewall still on) and usage went down according to them and according to the reporting utility I have access to for their numbers.Looking at the live numbers on our side ( when the LAN was connected) did not match their numbers. None of the logs I have match their numbers, but they said it is our issue.

So that did lead me to think that the firewall may be silently dropping data that isn't valid and that we may have a DOS or junk getting thrown are way due to bad equipment,or maybe it is invisible network traffic, but how do I prove it? No Firewall logs, no way to monitor the data incoming on firewall in live time, all I know is that the data is not entering our LAN.

Thank you for all the help I guess I feel a little better that I am not insane and I was accurately logging the data as you seem to point in the direction you think my data is valid. I guess I'll call ATT and see if they can help me figure this out, this time though I have data, lots of it to shove down their throats when they say it is us.

Alex

You could replace the firewall with a linux box and run diagnostics again.
That way you might figure out what's happening.

Only problem is I don't have a T1 interface card, or a T1 modem (if that's what it would be called) and I can guarantee the company won't spend money on one. I did speak with ATT and they told me they still show the PPP connection as maxed out on the T1 line for inbound traffic. I asked if there was a way for them to see that traffic and they said "Yes" that is if we had their router/firewall but alas we own our own equipment. I curse Juniper and NetscreenOS I have better logs and stats on my dlink lol. Seriously though for the cost the company paid for it, and the cost they want on the renewal for the services you would think they would at least have some sort of monitoring utilities you could run from the telnet or SSH admin connection, but they don't.