[Xapian-discuss] minor problem

[Olly Betts]

On Sun, Dec 23, 2007 at 11:31:17PM -0500, Michael A. Lewis wrote:
> Thanks for the response Olly. My indexing code appears below. A note
> about the speed. It was this slow (at least to the naked eye) even
> when there were only a couple of hundred documents. After this code,
> the child process which contains this code just exits.
>  
> try {                        
>         Xapian::WritableDatabase database(dbname, DB_CREATE_OR_OPEN);

The reason why this runs slowly is that you are reopening the database
for every single document you're indexing.  If you just open the
database once and keep it open, this will run a lot more quickly.  Also
you'll be able to add new documents in batches, which is a lot more
efficient.

The reason that this is slower with 1.0.x than 0.9.x is that flint's
locking code is different to that used by quartz (this was changed to
eliminate the problem of leaving stale lock files behind if an indexing
process was killed).  Part of the extra overhead will be because we now
call fork(), but in this situation there's also the overhead of the
lock-holding child process needing to exit (releasing the lock) before
the database can be reopened for writing.

Cheers,
    Olly