[Xapian-discuss] Future of Xapian (long)

[Olly Betts]

On Fri, Jun 18, 2004 at 04:47:54PM +0100, Charlie Hull wrote:
> a. Testing and fixing to make Xapian work in a multithreaded environment.
> Richard is currently helping to track down some threading problems, but this
> work could be expanded into a rigorous set of tests to check Xapian's
> threadsafety.

Xapian's approach to thread safety is to make it somebody else's
problem!  This was a decision we made several years ago.

We used to have lots of fine grained mutex locks around methods, but
they made the code harder to follow and harder to maintain.  It was
also hard to be sure that we were protecting everything we needed to.

The locks also made everyone incur a performance penalty (albeit
probably minor) for something which pretty much nobody actually wants to
be to do!  Typically, you'll either have one Xapian thread, or multiple
Xapian threads but each with its own Database object.  In either case
the locks aren't needed.

The thing we realised was that people using threads probably don't
require to be able to call methods of one object from several threads
concurrently.  And if they really do, it's not hard to add a mutex to
marshall the accesses.  By doing the locking yourself, you get to choose
the trade off between locking granularity and overhead.

So provided Xapian doesn't use static data or non-thread-safe library
calls (strtok, etc), there's not really much to do here.  A test along
the lines of Eric's example should give pretty good coverage, and it's
hard to be completely rigorous as such problems are highly dependent on
thread schedulding.  To be rigorous, it's better to take a code audit
approach - look for static variables, calls to strtok, etc and check
that any you find are OK.  Also check static methods are reentrant.  I
did a quick check of backends/quartz and couldn't see any static
variables there.

Cheers,
    Olly