[Xapian-discuss] flush problem

[James Aylett]

On Sun, Jan 20, 2008 at 03:30:04PM -0500, Michael A. Lewis wrote:

> The main thing the system seems to be doing is IOWAIT (17-68% is the
> approx. range I'm seeing over a few minutes).  The only appilcation
> running on this system is the XAPIAN code. At the time of the
> insert, no other processing is searching or inserting. Basically,
> only the flush code is running.

I'm assuming your data and database are on the same (logical, RAIDed,
volume). That will make it harder to see what's actually going on
easily, of course.

It's interesting that while you had high iowait, the process itself
wasn't always exhibiting it (from your previous posting of top
output). I can't remember with linux whether iowait sometimes isn't
attributed to the process, though.

> The getdatabase code is as follows:

[snip]
 
As you say, that's pretty simple; unless something else is messing
with that hash, it should do the right thing. (I assume you've checked
that nothing is messing with dbname.)

> In my previous posting I sent the output from a top command while it
> is flushing (which it currently is doing). Appears to be using 2.8gb
> of memory with 1.1gb free.

What does iostat give you on the logical drive during flush?
(Erm... iostat -x 2 2 on linux may be more helpful, taking the second
output block.)

> It's pretty simple. The average document length is about 300k of
> standard english text.

2.8GiB / 300KiB ~= 10000, which is what I'd expect if your manual
flush weren't actually doing anything. That's still worrying, though -
what happens if you do a single run, flush, and exit? Do the documents
get written correctly to the db?

It may be worth profiling your drive with bonnie or zcav or something
to get some figures about the logical volume. If they are unexpectedly
low, there's a different problem (I doubt that's it, but it's worth
eliminating).

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org