[Xapian-discuss] How FastMail is using Xapian with Cyrus IMAPd

[Bron Gondwana]

I've been meaning to make a blog post out of this for a while, but I haven't done it yet, and since I'm pushing to get Xapian to include our patches, I figured I should actually write something up to show how awesome Xapian is for us, and how it's working on our systems.

Cyrus IMAPd is written in pure C - Xapian is the first piece of C++ that's been included.  We knew we needed a search engine, and at first we were looking at embedding Sphinx, but the single server model worried us.  We wanted to have a search database per user for modularity.  That meant that loading the database and writing a new message's fields into it had to be super-fast.

We also wanted snippet support, which Greg Banks (the programmer working on that feature) added as a patch to Xapian.  I've attached an image to show the cross folder search results in our interface using that feature.

We built a C API wrapper around the Xapian features we needed, and tried to run in production.  Unfortunately, the IO hit from indexing new messages in real time was too much - our server couldn't cope.

Luckily, Xapian supports searching from multiple databases at once, so we came up with the idea of a tiered database structure.  We have 4 tiers at FastMail, though we don't actually use the 'meta' one (SSD) at the moment:

* temp
* meta
* data
* archive

So new messages get indexed to tmpfs in a small database in the temp tier.  A job runs every hour to see if tmpfs is getting too full (over 50% of the defined size), in which case it compacts immediately, otherwise a job compacts the temp and meta tiers down to the data tier every day.

Once a week, we compact all the tiers except archive back to the data tier.

The archive tier stays untouched unless we move a user, or as a manual operation, where we compact everything into a single archive.

Both external locking (see my other post a minute ago about write vs read and DatabaseModified errors) and the compaction logic are managed via a separate file called xapianactive.  The xapianactive looks like this:

% cat /mnt/ssd30/sloti30t01/store23/conf/user/b/brong.xapianactive
temp:264 archive:2 data:37

The first item in the file is the writable index - all the others are read-only.

These map to paths on disk according to the config file:

% grep search /etc/cyrus/imapd-sloti30t01.conf 
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive

(the 'default tier' is to tell the system where to create a new search item)

So based on these paths, we find.

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
3328	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.264
1520432	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.37
3365336	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2


So I haven't compacted to archive for a while.  Let's watch one of those.  I'm selecting all the tiers, and compressing to a single tier.  The process is as follows:

1) take an exclusive lock on the xapianactive file
2) insert a new default tier database on the front and unlock xapianactive again
3) start compacting all the selected databases to a single database on the given tier
4) take an exclusive lock on the xapianactive file again
5) if the xapianactive file has changed, discard all our work (we lock against this, but it's a sanity check) and exit
6) replace all the source databases for the compact with a reference to the destination database and unlock xapianactive again
7) delete all the now-unused databases.

Note that the xapianactive file is only locked for two VERY SHORT times.  All the rest of the time, the compact runs at the side.

This allows us to only ever have a single thread compacting to disk, so our search drives are mostly idle, and able to serve customer search requests very quickly.

So here goes:

% time sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z archive -t temp,meta,data,archive -u brong
compressing temp:264,archive:2,data:37 to archive:3 for user.brong (active temp:264,archive:2,data:37)
adding new initial search location temp:265
compacting databases
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3.NEW
renaming tempdir into place
finished compact of user.brong (active temp:265,archive:3)

real	4m52.285s
user	2m29.348s
sys	0m13.948s

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
368	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.265
du: cannot access `/mnt/i30search/sloti30t01/store23/search/b/user/brong/*': No such file or directory
4614368	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3


....

And there we go :)

Bron.


-- 
  Bron Gondwana
  brong at fastmail.fm