I am starting to get bottlenecks in the loading of webpages via squid.
It is not the actual playback of video that has performance issues, this is very good. However, when a webpage is first requested, it is the initial response that takes more than a few seconds to load the page.
I am wondering how to troubleshoot the issue as I don't know whether it is squid, the dns lookup, the number of users, RAM, or the bandwidth. Bandwidth seems to be the least likely since the server has a high bandwidth and video streaming appears to have no problems.
I do not have caching turned on yet, which would certainly help because I do not want to cache flb, mp3, etc. files and the last time I turned on caching the performance was actually worse. The slow loading of pages has only just beguin to creep into the performance, it wasn't an issue before.

I don't know the best way to confirm this, but that does sound like DNS. Have you tried using dig to examine the time that a lookup takes? And whether the first lookup is rather long compared to subsequent ones. (Are you just using the standard stub resolver (is the first server slow?) or do you have a cache?)

Mind you, you can get exactly the same effect by having a non-used IPV6 service in first and always waiting for that to time out before doing anything useful.

... RAM ...

That won't be the problem if no swapping is going on, so may be easy to eliminate.

when you get there, you might want to look at this.

Didn't seem to make much difference:
Could I add openDNS to squid somehow to improve the p[rocess?
At the moment I am restarting nscd every 6 hrs in case something happens - I'm sure this disconnects a few users.
I also have BIND on my server.

Not sure what the standard stub resolver is. I thought the cache was handled by nscd?

the standard stub resolver is the entry in resolv.conf that just says 'go to this other computer and that computer will get a name resolution for you'. The alternative is to be using something line BIND/DNSMASQ/DJBDNS/PDNSD as a caching nameserver.

Based on the evidence that you give, your lookups are not lightning fast, but then they are not disastrously slow, either (';; Query time: 69 msec', ';; Query time: 92 msec'). This probably isn't then the cause of this problem, and you'd want to do more testing, but the fact that the second resolve isn't a whole load faster than the first, and both are on the slow side, suggests that there is no local caching going on.

I'd guess you could get down to a half or a quarter of those figures with, eg, DNSMASQ/DJBDNS/... but as there is another problem, this isn't the place to start.

...not massively...a local cache (local to wherever squid runs) will be your best thing, and you can choose where to hook up your local cache. You can benchmark the external sources of lookups, but they often the top ones are quite similar, and it doesn't now look as if DNS is you biggest problem.

Does this problem exist when video streaming is not happening? It is possible that video streaming is eating all of the bandwidth.

The RAM/swapping issue ought to be easy to eliminate; check that there is no traffic to your swap when things are slow.

Then check that the memory usage of squid itself is sensible; is the video streaming using all of squid's memory allowances?

I have BIND running on the server but I turned nscd off since someone else said it made no apparent efficiency improvement. Worth noting that /etc/named.conf is just an empty file at present.
I added dns_nameserver and the opendns IP address to the squid.conf and restarted...no difference.
It is difficult to tell as there is always some video streaming happening on the site so I can't be 100% sure about an answer to this. However, I have much more bandwidth available than is being used as the server should have 100Mbit/s

How can I check the RAM/swap?
Squid mem usage is set to defaults in the squid.conf
I am trying to figure out how to use oprofile but no success yet...

These are the results from top:

.....