Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I've been working with Apache & MySQL for quite a while, but I've recently moved to a virtual dedicated server and I'm getting some bottlenecks. Support from my provider is not exactly satisfactory. They first said memory. Then I stopped a bunch of processes, and now they're saying bandwidth. I checked vmstat and found some interesting stuff.
Here's the setup:
CentOS
Apache2
MySQL 4.2ish
PHP5
I used top for a while but found nothing too bad. I got 256Mb of RAM down to 17Mb a few times, but no swapping. I had some rather large includes that didn't have to be included, so I cut those out. I also stopped some java crap, tomcat, and cpanel which also gave back like 30-40 MB.
Then they come back w/ the bandwidth (10Mb/s) story. I tried to monitor this, but iftop doesn't seem to be as intuitive as it should. Assuming that might be it, I capped dl speeds to 100kb/s. I don't have that many users, and I don't host that many large files to be honest.
Anyway, the problem persisted. So, I started browing, tailing the httpd access log, and running vmstat. Looks like cpu hoggage is the error [last field -- wait time]:
So, I think it's fair to say that it's the CPU. As before, this is a virtual dedicated server. I'm trying to find out if CPU usage is throttled like memory & bandwidth. In the case that it is, that still points to a problem of my own doing (or bad config on their side). In that case, what other tools can I use to figure out what the hell is going on? During the above example, /var/log/messages prints out nothing, and httpd.access.log nothing until the bottle neck finishes. Please don't tell me to get a fully dedicated server (I know that's a very likely option at this point) but I'd rather avoid the cost and for the love of God... the work in migrating.
There was an incident where the machine was unreachable (ssh or http). They had to power cycle it. They said it was due to "lack of memory" which sounded pretty lame. It's not happened again and I've been running the same stuff. Only time I've seen something like that happen would be something in a infinite loop, or some key component crashed.
Anyways, cheers
EDIT: This perhaps should've gone somewhere else? It was originally going to be Apache-centric, but I don't think that's the issue.
Last edited by GaijinPunch; 12-13-2007 at 12:54 AM.
The wa field is IO wait time. It looks like an average 50% wait time for IO. That seems like bandwidth--the low bi/bo figures indicate it is not disk*. It is also informative that context switches (cs) double, and buffer allocations double as well, which fits with a bandwidth limit.
*Actually, I'm a little suspicious of the 0's in [sb]/[io] and in. Perhaps being in a virtual machine is messing those numbers up.
5. Run dmesg and look for the dreaded __alloc_pages() : 0-order allocation failed (gfp=0×1d2/0) message – this is a sign that the kernel has hit its memory limit and has had to take drastic action to keep the system running.
This would fit with your ssh and http being unavailable that one time.
Which virtualization solution are you hosted in? (Xen, VMWare, etc.)
IMHO it ain't CPU - it's disk. That iowait% is the percentage of time the CPU was idle, but I/O was outstanding. Probably all your I/O now goes to one "container" - maybe on the same (real) disk as other clients.
If you were short on CPU, when your (virtualized) client got dispatched, it'd go nuts trying to catch up - the "using" numbers don't show that.
Try installing the sysstat package - for sar and iostat.
In the interim, maybe add the "-d" switch to your vmstat runs.
Hey guys, thanks for the replies. Sorry I'm late getting back.
I killed some crappy processes, freeing up some 120MB of RAM, and it's still persisting. I have noticed the last few times that the bottlenecks happened, there were 50 httpd processes running. I have dropped MaxClients down to 40 (w/ 25 threads per process) but it still happens.
Quote:
Run dmesg and look for the dreaded __alloc_pages() : 0-order allocation failed (gfp=0×1d2/0) message
I guess due to the virtuality, I get no dmesg.
Quote:
Which virtualization solution are you hosted in? (Xen, VMWare, etc.)
Not sure. I would have to check. It's whatever GoDaddy offers. I'm thinking of just going dedicated. It seems I get 400+ visitors (that's unique visitors) daily, and that's not counting webcrawlers (well, most of them anyway). I'm not sure if that warrants my own server or not. I guess that would depend on how much each person does, which I should probably log a bit better. Some of the pages load other pages (from other sites) and parse them... I assume that doesn't help too much either. I could probably smarten up the code a bit, but something tells me it's not going to make a ton of difference.
Quote:
Probably all your I/O now goes to one "container" - maybe on the same (real) disk as other clients.
In the interim, maybe add the "-d" switch to your vmstat runs.
More frownies:
Your kernel doesn't support diskstat (2.5.70 or above required)
Another point for going fully dedicated?
Last edited by GaijinPunch; 12-18-2007 at 06:32 AM.
Here's an update.
-Web server was crucified
-SSH okay
-top not bad (below)
-vmstat (below)
-apache access log (chugging along.. showing almost no activity likely due to stall)
-apache processes (40: max)
In this instance, I had to restart apache. Once I did, I had many connections, and only one httpd process.
The problem with iftop is I can only see as many connections as can fit in my terminal. Anyone know of a better network logging tool? One that can tell me a bit of historical data (max connections hit, when, what port, etc)?
I'm going to power cycle in a minute. I borked up the email server somehow and I'm not paying $75 per 30 minutes for some nimrod that knows less than me to tell me how to fix it.
Seems to me you need to be looking at some "sar" output - do you have sysstat installed and active ???.
Not sure what data you get in 2.4, but you need to be able to correlate problem time and data.
May not be a lot of help if it's an (internal) Apache issue, but at least you'll have plenty of data to check.
I will check those.
In the meantime, I installed a mod that will limit connections per IP. I noticed yesterday that some ass clown was either slurping the entire site, or just downloading the few files I have with as many connections as he could. This of course, doesn't help the situation. I will continue to watch.
EDIT: That mod doesn't seem to be working. The same guy is pounding my site. I need a mod that will beat the piss out of someone based on IP.
Last edited by GaijinPunch; 12-18-2007 at 05:51 PM.
Yeah I would say that the guy who was creating a ton of connections was probably the problem with the bottlenecking. I had a similar issue with my own server testing out proxy stuff I had it open on a non standard port for just a short while none the less after only a few min of being online i had people using it to browse the internet and slow down my own requests to the server.
Yeah I would say that the guy who was creating a ton of connections was probably the problem with the bottlenecking.
I got the mod to work it seems. However, it seems he's now connecting a maximum of 3 times per connection. whois show's they're from the same company. I banned the entire range. Let's see what disaster he comes back w/ now.
Hmm....
when I use iftop, it shows that the offending site is connecting on *THEIR* http port, and my port 53667, 54037 etc. Wouldn't that seem like I'm making the connection? I use CURL to get some pages... but not that many. Hmm....
-GP
Last edited by GaijinPunch; 12-19-2007 at 12:19 AM.
Im sorry that I don't know more about this stuff, however, I had problems with what I thought was my isp. My internet connection upload and download had greatly slowed.
I powercycled my router and all worked great again.
For anyone following this, I'm still stuck on this problem. I look at iftop, but it really doesn't tell me much other than I have a lot of connections. However, when it bottlenecks, that number gets cut down to 10 or so (hardly enough to pound the server).
I'm told I only have 256MB, which is the problem (according to my provider) but I've never seen the machine swap.
Can anyone else think of some other tools I should look at? Maybe I've got some basic apache configuration cocked up. Is there a nice guide that tells me what type of machine I should be using for the amount of traffic I get? I know it depends on a lot of things, but still.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
