Linux/Unix - Squid weirdness

Author

Squid weirdness

ccieToBe

2002-10-08, 9:15 pm

One of my Squid servers seems to fubar whenever it has trouble connecting to many sites for an extended period of time. The server itself, which is running a few other services is fine, but something's happending to Squid. This has happend 3 times. Here's what's going on:

1. The Squid server has trouble accessing requested sites over an extended period of time for some valid reason. This last time (last week), it was due to our not being able to access certain sites living in Worldcom's network.

2. When whatever was going on is resolved, Squid seems to go fubar. This server also runs Samba, mpd-netgraph, dhcpd and and a handfull of security releated apps. Squid seems to be the only app that's affected.

3. If I do

/usr/local/etc/rc.d/squid.sh stop
/usr/local/etc/rc.d/squid.sh start

then the problem is resolved.

This is a FreeBSD server (old 4.6-stable snapshot) running Squid 2.4. This is actually a very overpowered server (replaced an underpowered one a few months ago). When the process goes fubar it works for most (maybee 3/4) sites slowly, and times out on others. During these periods a lot more info is logged then normal. Here are a couple of messages that were repeated often:

httpAccept: FD 10 (53) Software caused connection abort
comm_accept FD 10 (53) Software caused connection abort
idnsCheckQueue ID 5200: giving up after 33 tries and 301.6 seconds

Any ideas? I'm not sure what to think. The cause seems obvious, but not the fix. This server's running with very close to a default config. I changed a dozen or so parameters.

Mr. Linux Guy

2002-10-09, 6:26 am

Check your cache.log for extra error messages. Squid dumps some messages to /var.log.messages, but most go to the cache.log file. If the messages there do not seem to be overly informative, you can try strating squid with debugging enabled, like so:

# ./squid -k debug

This causes every debug() statement in the source code to write a line in the cache.log file. You use the same command to undo debugging.

If this doesn't shed any more light on the prob, you can try getting a core dump to see if that could give you some information. You can usually generate one like so:

# limit core un; ./squid -NCd1

Then you can get a stack trace with the GNU debugger (gdb).

Actually, this sounds like it might be a memory problem with squid. What do you have your cache_mem in squid.conf set to? You may want to try increasing this value. This parameter specifies how much memory to use for caching "hot" replies. Squid is a memory hog and not having enough set may cause problems.

In the meantime, you may want to use cron to schedule a daily stop and restart of the squid service as this should clear the memory out. The actual "abort" message means that a client aborts their requests before a page fully loads. Extreme network congestion might also cause this, so you may want to check that all is OK on the net. The "giving up after xx tries" often means that a DNS query timed out, and that generally is not a squid problem. Since all is well after you restart the service, I am guessing that it is a memeory error. Try monitoring your processes and memory conditions to see if this is actually the problem. You can write a script that can do this for you on a periodic basis in order to correlate the hang time with system conditions.

ccieToBe

2002-10-09, 10:17 am

cache_mem is set to 64MB (the default was 8MB). This server has a gigabyte of RAM, so I don't see any harm in doubling this - I'll give that a shot.

I'll try restarting squid in debug mode later on today (the server's in use right now and I don't feel like getting any phone calls telling me "the Internet's broken"

).

Other then when the server's starting up, I've never seen the processor or memory utilization even reach 30%. I checked this before restarting Squid and don't remember what they were, but do remember that they looked typical.

Of course, restarting Squid brings that processes's memory usage from 15%-20% to 0%, so you may have something there. Could it be that unfulfilled requests hog memory? What you're saying makes sense, but I'm not sure how not being able to get to certain sites for some external reason would trigger this.

Thanks Mr. Linux Guy.

Mr. Linux Guy

2002-10-09, 10:23 am

Yeah, I see what you are saying. I am not sure the log requests that you got are necesarily related directly to your problem, as those are quite normal log entries and usually do not indicate anything odd going on. If you turn debugging on, you may get more informative error messages. If, on the other hand, squid sort of maxes out it's memory, or at least does not manage its memory as it should, there may be little room left for it to do its job and then you might get a stall or even a crash. I have heard of others having this problem, and it still sounds like a programming problem . . . squid it sounds like is not managing its memory properly, so if possible i would try increasing its quota. Restrating the service would flush the memory, and that would be pretty easy tomanage, but its at best a workaround. I will see if I can find any more info about this and let you know what I find.

ccieToBe

2002-11-15, 3:23 pm

So far it looks like increasing the cache_mem parameter has fixed the problem.

Mr. Linux Guy

2002-11-15, 4:11 pm

That's great. You may want to make sundry notes about things like this and compile it into (the horror!) *documentation*. I know it's a pain in the arse, but all the same, it can come in really handy when you run into a problem, to say nothing of when someone else has to come in and take over where you left off when you get offered a more lucrative position.

I know I also tend to solve the same problem a number of times because I forget how I did it the first time; if it doesn't recur that often but was an issue, a note in a documantation folder might save pain in the future. Just a thought. Glad the prob seems to be fixed.

ccieToBe

2002-11-15, 4:57 pm

That's something that I've been neglecting up to this point. I heavily comment and backup my scripts, and had to take inventory a few months ago right before the acquisition, but other then that, it's all in my head.

Now that I'm no longer the only in house Unix talent I probably should start making more notes.

Mr. Linux Guy

2002-11-16, 5:21 am

Yeah, its one of those things that most of us keep putting off until we realise that it would have been real handy if we had it already! After a system reaches a certain critical size, it usually becomes necessary for the docs to be made available even for one's own use, as I mentioned above, for problems that are serious or annoying but don't crop up all that often. As for others, it becomes even more important for maintaining someone else's system or their code as well. You could get a little PC to act as an Intranet server and add the docs there as you do them so that all the techs can have full access to them as they are made available, although offline docs are also quite useful.

Another pertinent point: It is *much* easier to do this when your system is still of manageable size that once it grows to a fairly complex operation.

Yet another point: Not usre if your organisation does this yet, but if any system auditors (am not talking about IRS auditors here, but people who check your institution for compliance with 'standards' business procedures and so forth) come by to check out your operations for any number of reasons or if you need to bring in a consultant for temporary work on something, it helps a lot. Lack of system docs during an audit of some kind can get you cited, but not sure if this is as common in the private sector as it is in education.

ccieToBe

2002-11-16, 12:16 pm

We already have a web server setup for the Intranet, so it would be easy to setup the docs there, then include them in the daily backups so that I have an offline version as well.

I don't think there's much chance of an audit. The BSA mafia might come through, but I've managed to bring our software that requires licencing down to a handfull of programs running on the workstations that are easy to keep track of. Most of our systems are based on OSS

The company that bought us out is fairly large, but I don't think they do any internal audits.

Keeping good docs for consultants sounds like a good idea.

Sponsored Links