[Xapian-discuss] flush problem

[Michael A. Lewis]

I am seeing the following from TOP:
 
top - 14:53:37 up 17 days,  4:24,  1 user,  load average: 2.89, 3.09, 3.08
Tasks:  77 total,   1 running,  76 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us,  1.0%sy,  0.0%ni, 17.5%id, 75.9%wa,  0.2%hi,  0.2%si,  0.0%st
Mem:   3761652k total,  3635644k used,   126008k free,     3004k buffers
Swap:  9213268k total,  1311104k used,  7902164k free,   551384k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                     
 3159 root      18   0 2907m 2.8g  636 D   12 78.5 866:00.14 ftinsert 
 
With FTINSERT being the process that is my insert task. Strangly, it is not using a great deal of CPU top, I usually have to wait for a little while before it is the top task in the list (sorted by CPU usage). This is a 4gb dual core system running a fast raid array. Again, this is only with 560000 documents or so.
 
--Michael
________________________________

From: xapian-discuss-bounces at lists.xapian.org on behalf of Jesper Krogh
Sent: Sun 1/20/2008 2:46 PM
To: xapian-discuss at lists.xapian.org
Subject: Re: [Xapian-discuss] flush problem



Michael A. Lewis wrote:
> I am having a problem with flushing a database. I am adding N records
> to the DB (which amounts to 1 - 2000). At then end of the run, I
> issue a flush() call. The problem is that the flush call never seems
> to do anything. Every 10000 additions to the database and the library
> performs a flush (which can take up to 3 hours on a 560,000 document
> database) as if my flush call was never performed.

Not that I have a solution, but I have a similar problem with my Xapian
database. (doccount 8millions) flushtime is fairly long (over 10 minutes
on a 16 SAS disk array for 1000 documents added) and monitoring vmstat
(and top) I can see that it neither saturates 1 cpu or anything near the
block input/output that the disk can deliver (uses around 5MB/s in
block/in and out), viewing "top" only around 8-12% IO wait.

All of above is measured when Xapian is "flushing".

Still running xapian 1.0.4 (with perl-bindings)

--
Jesper

_______________________________________________
Xapian-discuss mailing list
Xapian-discuss at lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss