[Xapian-discuss] indexing strategy for "near real time" indexing

[Jarrod Roberson]

I am working on a proof of concept real time email indexer using xapian.
This is for HUGE volumes, think ISP level.
I have to come up with a strategy for indexing the messages as they come in
as near real time as I can.
I am considering indexing into many databases based on time and / or size,
and then trying to xapian-compact them
together at the end of the day, and start over. The single writer limitation
is what I am trying to address.
Anyone have any suggestions about what might be a good place to start?