Too many open files

[nickpick]

Hello,

On my Plesk 8.2 / CentOS 4.2 box
this morning i found Apache dead..

The error_log was empty and when I try to start apache I get following:

[error] (24)Too many open files: could not open transfer log file /var/log/httpd/access_log.
Unable to open logs

Rebooting does not help.

Anybody out there does recognize this problem?

any advice would be welcome.

[nickpick]

Need to add following to my previous post:
I just temporarily moved away from /etc/httpd/conf.d the Plesk file: zz010_psa_httpd.conf

Now apache just starts fine and the non-Plesk websites are running perfectly.
So now we know what creates the problem: the Plesk virtual hosts config.
And i guess there are too much virtuals configured, or something like that.
I currently see over 1300 virtual containers.

[breun]

See the Knowledge Base: http://kb.swsoft.com/en/260

[marcbote]

Hi,

since you have Plesk 8.2, you can use the new feature SWsoft introduced on this release of piped logs.

This will avoid you to recompile.

Here you have the step by step in Plesk manual:

http://download1.swsoft.com/Plesk/Plesk8.2/Doc/plesk-8.2-unix-administrators-guide/44461.htm

[nickpick]

Thanks, marcbote.

That is something that is usefull.

All stories about "recompiling apache" are a complete HORROR story and are UNacceptable.

I am sad to find out that Plesk support also recommends to recompile apache, without referring to the article you point at.

Even worse is the fact that the Plesk GUI does not automatically do this simple "setting" when it detects that there are over 900 domains when upgrading !

That is what pisses me off: They know about the problem. They know how to fix it. They leave the bug in...

I will check out the "piped logs" and post results in this thread.

[marcbote]

Hi,

well, in my point of view that's not a limitation of Plesk itself, i really don't know why RHE/centos/fedoras/debian.....put this little FD descriptor, freebsd afaik don't have it so little.

About piped logs, it's a good solution indeed, and it's more than great that now Plesk supports it also to don't have to recompilate, but in my point of view, more than avoid recompilation best thing it's avoid problems having special packages outside distribution (a lot of people after having that fixed runs a yum upgrade or apt to update packages and overwrites that...and then due this..another downtime)

[nickpick]

First I need to correct Plesk's manual into the following:


Enabling Piped Logs for Web Server to Avoid Disruption of Web Services
If you are going to host more than 300 domains or Web sites on your server, you need to switch on support for piped logs in the Apache Web server. To do this, follow these steps:

1.Log in to the server shell.
2.Issue the command mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa -e "replace into misc (param,val) values ('apache_pipelog', 'true');"
3.Issue the command: /usr/local/psa/admin/bin/websrvmng -v -a


Again: why isn't there just a setting in the Plesk GUI to perform this?
Can anyone conform if this problem also exists in Plesk 8.1 ?

[nickpick]

Originally posted by marcbote
Hi,

...but in my point of view, more than avoid recompilation best thing it's avoid problems having special packages outside distribution (a lot of people after having that fixed runs a yum upgrade or apt to update packages and overwrites that...and then due this..another downtime)

There is exactly the point why recompiling is a horror story. I manage dozens of Plesk servers on RH distributions. That is why I ONLY use the standard available packages. Managing all the custom-compiled boxes would simply be impossible. Especially when yum, apt or up2date starts running automatically.

Thanks for your usefull posts.

[nickpick]

I keep updating my post:

Just enabling the piped logs did not fix the problem at all.

According to Plesk there is a limit of 600 domains on Plesk 8.1 and 900 domains on Plesk 8.2 when you enable piped logs (?).

Since this server has over 1300 domains configured (mostly redirects) there is still the problem.
While with Plesk 7.5.4 there was no problem at all !

So to overcome this issue I have increased ulimit. But that can only be done on shell level with: ulimit -n 2048 (remember the number can be different in other situations)

To make this active after a reboot, or when restarting apache in Plesk GUI. I did following "fix":

I added "ulimit -n 2048" as the first line after the comments in /etc/init.d/httpd

I know this is not a nice solution. But it will work for now.
Again: recompiling apache with higher fd_setsize is NO option.

I think in the Plesk file /etc/httpd/conf.d/zz010_psa_httpd.conf there is the problem.
In every virtual container is included following since Plesk 8.1:
CustomLog /var/log/httpd/access_log combined
Even for all redirected domains.

Is there anyone in this forum who can shine a light on this with some more technical detailed info?

[igorx]

The too many open files problem is because of the CustomLog directives in apache conf, as you mentioned.

I've been trying several solutions for many day now, without complete success.

Raising the open file handlers with ulimit enables starting apache, but does not solve the horde problem.

I decided not to recompile php with raised FD_.

My solution is to comment out the CustomLog lines (it has no effect the domains log files and so the statistics).

This solution is ugly, but works. It comments out the CustomLog lines.

1. To do this, create a file /usr/local/bin/pleskcorrect with 755 rights and with the following content:
#!/bin/bash
awk '{ sub(/CustomLog/,"#CustomLog"); print }' /etc/apache2/conf.d/zz010_psa_httpd.conf > /tmp/plesk_awk;mv /tmp/plesk_awk /etc/apache2/conf.d/zz010_psa_httpd.conf

Change the apache config file path sections of the command according to your system

Of course awk must be installed.

2. then place the /usr/local/bin/pleskcorrect command in your apache startup script (eg: /etc/init.d/apache2 or httpd).

Plesk updates the virtualhost config file and removes the comments every time a domain is created or deleted. But after then it reloads apache, but the startup script comments out customlog lines again.

[nickpick]

Thanks igorx.

Thanks for sharing it. I don't think it is an ugly solution. It is a nicely done Plesk-bug FIX.
For now I have stopped upgrading servers to 8.1
Plesk must fix this.

We need to complain about this to Plesk, because it is them who put something useless in the system that causes this problems.

I know that Plesk tries to blame vendors for this by claiming that they compiled apache with too small number of file descriptors.
But the vendors have been doing so for many years since RedHat 7.2 (that is as far as I can trace back).
We never had problems, untill upgrading to Plesk 8.1

So everybody must complain about this to his Plesk sales representative. That is possibly the only way Plesk will listen to us.

I am using 35 licences of Plesk Unlimited. But now my licences are all downgraded to limited licences. This counts for every owner of a Plesk licence.

[atomicturtle]

How is it a plesk bug? Apache, php, etc come from the vendor, and Apache, not sw-soft. How are they supposed to fix something they have no control over?

[nickpick]

Plesk wants MONEY for their package and Plesk say they support RedHat, CentOS, etc.
Well, they do NOT.
Because they do NOT properly support RedHat distributions, etc.

This so called limit has always been there in this distributions. Plesk should be (and IS) aware of that and should act properly.

They should even warn for this while selling UNLIMITED licences.

You should read the entire thread and try to understand before posting.

[atomicturtle]

I did, Im also the guy that figured out how to fix this, and the one who advocated to move PSA to the RPM design in the first place.

Nothing in Plesk keeps you from being able to do it, its entirely dependent on the Operating System. You wouldn't blame sw-soft because your kernel doesn't support your hardware, so why would you blame them for something else that they have no authority over?

[marcbote]

freebsd for example don't have such limitation....you choose distribution, each one have it's good, and bad things.....

It's up to sysadmin the reasons of choosing one or other distribution, even there is people who chooses ones wich shall be compatible with the rpms of attomic...why not...it's a reason, aswell as i like debian....or this morning my heart bits in the tone of.....

I agree with you at 100%, plesk shall not preven things, but like with piped logs...support features from oses, packages aplications handled to maximize them.

[tedocweb]

Please stop the flaming in this forum. It's useless.

Turtle, I don't care if you think you advocated Plesk to move to RPM design or not. I think it is only in your head.
Now you also say that you figured out this solution?
Then why post here as "igorx" and later play advocate of the devil as atomicturtle?

Thanks anyways, because now I don't have to spend time to find a solution to this Plesk-bug.

I am glad Nick opened this thread so I instantly found a solution after my first apache went down calling out "too many open files" after upgrading to 8.2

I rarely come to this forum, because I am a Gold Plesk partner and always get great support from the Plesk support team.
Just not this time...Only thing Plesk support telling me was to recompile apache and pointing to the KB. Only alternative they gave me was to buy a new server with a Plesk licence and migrate 200 domains.

Yeah, right !!

Well my friends: my servers have run stable since Plesk 1.3 in 2001. And only now after upgrading to 8.2 there rises this problem.
SW-Soft CANNOT BLAME IT TO REDHAT.
I know about this limitation for years. Why not the big SW-Soft?

Plesk say they support RedHat. They took my euro's for unlimited licences. And now the licences seem to be limited to 900 domains.
That sucks! Don't try to convince me otherways.

The combination of solutions described here is good and will do untill Plesk comes up with a fix.

Yes Nick, I have already complained to my sales representative at SW-Soft in Germany, because I think this time they have really fucked up.

[atomicturtle]

Not sure I follow you there, but I'm not igorx if thats what you're implying. I am however a founder of plesk.

Your frustration is understandable, just misplaced. You're trying to make the system do something that it wasn't built for. Take plesk off the system, and try to do what you're doing, you'll have exactly the same problem.

[marcbote]

Dear tedocweb

Had you tried the solution of pipedlogs as described in the manual?? Had didn't worked?

I tried it in several boxes, where the amounght of domains growed and growed and worked like a champ, from down to up in a short seconds...

[igorx]

Piped logs has limitation on domain numbers.

I have 1300+ domains and piped logs is useless in this case.

[nickpick]

You might understand I got very frustrated by the kind of responses of guys like the turtleman and Plesk pointing to the knowledgebase.

So I called Ensim and they told me they have not such a problem. So I bought an Ensim Pro X unlimited domains licence.
I found my son willing to put the 1429 domains on it and the server is still running stable.

Thanks to Ensim. No thanks to Plesk.

[tedocweb]

Dear marcbote,

The activation of the piped logs did not fix the problems (I have over 1300 domains on this particular server).

So I did the nickpick-solution.
That helped and I got the sites up again.

I also tested the igorx-solution. That worked too.

I read that nickpick switched to Ensim...
That is not an option for me, because Ensim does not (yet) have a migration utility.

[atomicturtle]

What distro are you on? I can whip up some rpms.

[nickpick]

Hello Atomicturtle.

Sorry for the slow response. I needed to consume some holiday time :D

I use Centos 5.0 and 4.4 and RHEL4.2

[atomicturtle]

i386 or x86_64?

[nickpick]

up to now only i386

[jjmusicpro]

i am also haveing the to many open file requests.

I only have 25 domains, 10 of those are being used, but i have 600+ sub domains...

i did the quick uplimi -2048 fix.

but i want something that is going to work better.

i have plesk 8.1 on freebsd

[nickpick]

You could try the Igorx-solution in the first page of this thread:
http://forum.swsoft.com/showthread.php?s=&threadid=45895&perpage=15&pagenumber=1

He calls it "ugly", but it really isn't ugly, if you don't want to recompile apache.
The only best solution in this case indeed is recompiling Apache. I have to admit.

[jjmusicpro]

Originally posted by nickpick
You could try the Igorx-solution in the first page of this thread:
http://forum.swsoft.com/showthread.php?s=&threadid=45895&perpage=15&pagenumber=1

He calls it "ugly", but it really isn't ugly, if you don't want to recompile apache.
The only best solution in this case indeed is recompiling Apache. I have to admit.

is this something you can do? i have no idea lol

[breun]

Yes, see the knowledge base article I linked as my first reply to this thread.

[jjmusicpro]

Originally posted by breun
Yes, see the knowledge base article I linked as my first reply to this thread.

I PM you.

[nickpick]

Originally posted by jjmusicpro
is this something you can do? i have no idea lol

If you don't feel sure about this things, and you can wait, then just wait till the atomicturtle comes up with some rpm's.

[atomicturtle]

Its on my list, I havent forgotten about you guys. Just need to get some other stuff done in ASL before I can get back to this.

[jjmusicpro]

Originally posted by nickpick
If you don't feel sure about this things, and you can wait, then just wait till the atomicturtle comes up with some rpm's.

what is a rpm?

[atomicturtle]

Redhat Package Manager

[nickpick]

Originally posted by jjmusicpro
what is a rpm?

check this: http://www.rpm.org

[MaximumPig]

I am getting too many files errors when I run PEAR's Queue_mail class to send even moderate amounts of email to my host provider's smtp server. This PEAR class uses MySQL to store emails in a queue and provides a method to send batches from the queue off to the smtp server (I am using Godaddy which requires the use of their own smtp server for all outgoing mail from virtual dedicated servers). I believe, although I am not certain, that the PEAR class bypasses qmail entirely. In any event, even after just 100 or so emails, the whole process slows and dies and when I ssh in to see what's going on I get the dreaded too many files errors.

My questions are:
- which, if any, of the fixes in this thread might apply to my situation? I am at the moment only hosting 3 sites so this is not an issue regarding # of domains, but rather Fedora Core 6 and/or Plesk's default limits on file handles. Most of the discussion in this thread seem to relate to increasing the # of domains so I am wondering if these tweaks might more generally alleviate file handle restrictions.

- is there a better/less overhead-intensive way to send mass emails using PHP (and MySQL if necessary)

i tried raising ulimit -d to 10000 but that does not seem to help.

thanks in advance for any advice or help.

[jjmusicpro]

this is a huge problem and no one here seems help with this.

[HostingGuy]

Hello all,

We ran into a problem with the dreaded "Too many open files" error message in Apache which caused it to crash and left it in a non-startable state.

We fixed the problem by editing the /etc/security/limits.conf on a RHEL box, saving and restarting apache.

I hope this helps some one.

[jjmusicpro]

ok explain what you did...edited what.... what commands etc?

[HostingGuy]

we changed

* soft nofile 32768
* hard nofile 32768

[MaximumPig]

thanks for the info. is that intended to fix just the "more than 900 domains" problem or any problem that results from too many files?

[HostingGuy]

At this time we are not using piped logs, and have aprox 550 domains using plesk 8.2.1

We changed those settings in response to apache crashing due to "too many open files" error.

The settings we changed only change the number of open files the system will allow in both soft and hard limits, it doesnt have anything to do with any specific application such as plesk, apache, etc.

[modguru]

Any solution for this?

I have more then 900 domains, so piped logs wont work.

I tried to recompile but all that stuff seems to be a bad idea.

I am using CentOS 5.1-32

Is someone having recompiled RPM's that I could install?

atomicturtle where r u?

[breun]

I tried to recompile but all that stuff seems to be a bad idea.

Why does that seem to be a bad idea? I agree it is not so nice that you can't just yum update those packages anymore, but according to How do I enable Piped Logs for Apache Web Server (http://kb.parallels.com/en/2066): "If you have more that 900 domains, you still have to recompile Apache and some other system packages as described in article 260 (http://kb.parallels.com/en/260)". If it's the only solution, I don't think it's a bad idea. Or you'll have to get more servers.

[modguru]

Why does that seem to be a bad idea? I agree it is not so nice that you can't just yum update those packages anymore, but according to How do I enable Piped Logs for Apache Web Server (http://kb.parallels.com/en/2066): "If you have more that 900 domains, you still have to recompile Apache and some other system packages as described in article 260 (http://kb.parallels.com/en/260)". If it's the only solution, I don't think it's a bad idea. Or you'll have to get more servers.

I agree, I tried to recompile but this problem still continues.

[HostingGuy]

I had this problem and recompiled all of that stuff and my problem went away. its not elegant but it does work.

i would recomend using piped logs no matter what as that will reduce the number of open files as well.

The php packages that ART has are already a high FD build, but he doesnt do apache, openssl, curl, mod_ssl, etc so you will need to those.

Just go to http://isoredirect.centos.org/centos/5/os/SRPMS/ and download the sources you need.

Then run rpmbuild --rebuild rpm-here and see what happens. Most likely the first couple of times it will fail because you need to install some pre-requisites first, just use yum to install those and keep trying.

[modguru]

Thanks, will give it a try.