[Xapian-discuss] File Descriptors

[Arjen van der Meijden]

We have a server running over 4000 queries per hour at peak hours. It 
has never given us any problems with file descriptors etc.
It does run the queries via xinetd, so there is a hard-limit on the 
amount of omega processes that can co-exist, but I don't believe it has 
often, if ever, reached that limit. The machine allows 1024 open file 
descriptors, according to ulimit -n. The upper-limit is a bit higher 
though: 399767 (according to /proc/sys/fs/file-max). I have no idea why 
that value is so high, probably because it doesn't matter too much.

So I don't think you should be afraid to run into the maximum number of 
open file descriptors. When you run into that, your server will be 
extremely heavily loaded and you should take actions anyway.

Best regards,

Arjen

Richard Boulton wrote:
> On Fri, 2005-01-28 at 11:19 -0500, Mike Boone wrote:
> 
>>We are currently having problem with "too many open files" on our system. We
>>have a busy server that runs Apache, MySQL, and PHP on Red Hat Enterprise AS
>>2.1. The most "exotic" thing the system runs is Xapian 0.8.3, which is
>>accessed via the PHP-SWIG binding. I'm currently investigating the cause,
>>and I thought I'd check with this list to see if there was anything obvious
>>that had to do with Xapian.
>>
>>We're averaging something like 18,000 Xapian queries a day. Does each Xapian
>>query create a lot of open files? Is there anything special I need to do to
>>make sure those files get closed? Are there any special considerations since
>>I'm using Xapian via the PHP-SWIG binding?
> 
> 
> Each open Xapian database uses 5 file descriptors (and a 6th is used
> whilst the database is being opened, or when the database is being
> flushed).  This could lead to quite a large number of filedescriptors
> being used.
> 
> On Linux I believe there are two file descriptor limits - one is a hard
> limit set by the operating system (but is quite a high value).  The
> other is a lower, per-process limit, which can be changed.  Under bash,
> "ulimit -n" displays the current per-process limit, which may be
> modified if you have sufficient permissions (usually root).
> 
> All file descriptors used by a Xapian database should be released when
> the database is closed, so as long as there isn't a bug in PHP-SWIG
> which is failing to delete databases after use, there shouldn't be a
> problem that raising the limit won't solve.  I haven't looked at the PHP
> bindings in detail, but it looks like you are meant to explicitly delete
> database (and other Xapian objects) after use, so it would be worth
> making sure that you're doing that.
>