[Xapian-discuss] minor problem

[Michael A. Lewis]

Thank, Olly. After your last email this was my suspicion also.  I am recoding to keep the DB open. Unlike most Xapian Apps I've seen, this app is transaction based. Records are added one or two or 100 at a time. I'll let you know how it turns out.
 
-Michael

________________________________

From: Olly Betts [mailto:olly at survex.com]
Sent: Wed 12/26/2007 5:56 PM
To: Michael A. Lewis
Cc: Xapian Discussion
Subject: Re: [Xapian-discuss] minor problem



On Sun, Dec 23, 2007 at 11:31:17PM -0500, Michael A. Lewis wrote:
> Thanks for the response Olly. My indexing code appears below. A note
> about the speed. It was this slow (at least to the naked eye) even
> when there were only a couple of hundred documents. After this code,
> the child process which contains this code just exits.
> 
> try {                       
>         Xapian::WritableDatabase database(dbname, DB_CREATE_OR_OPEN);

The reason why this runs slowly is that you are reopening the database
for every single document you're indexing.  If you just open the
database once and keep it open, this will run a lot more quickly.  Also
you'll be able to add new documents in batches, which is a lot more
efficient.

The reason that this is slower with 1.0.x than 0.9.x is that flint's
locking code is different to that used by quartz (this was changed to
eliminate the problem of leaving stale lock files behind if an indexing
process was killed).  Part of the extra overhead will be because we now
call fork(), but in this situation there's also the overhead of the
lock-holding child process needing to exit (releasing the lock) before
the database can be reopened for writing.

Cheers,
    Olly