To: notmuch@notmuchmail.org
From: Olly Betts <olly@survex.com>
Date: Mon, 15 Mar 2010 09:29:35 +0000 (UTC)
Lines: 32
Message-ID: <slrnhprvfv.hu6.olly@msgid.survex.com>
References: <4b9dccc0.c6c1f10a.3671.44ec@mx.google.com>
	<20100315090401.GA29891@glaive.weftsoar.net>
User-Agent: slrn/pre1.0.0-11 (Linux)
Subject: Re: [notmuch] Notmuch performance (literally, in my case)
Precedence: list

On 2010-03-15, Hans Dieter Pearcey wrote:
> On Sun, 14 Mar 2010 22:59:28 -0700 (PDT), Ben Gamari wrote:
>> Notmuch is using xapian 1.08-1.99karmic from the Xapian backports PPA, which
>> I believe includes the recent database update optimizations.
>
> As far as I know, it doesn't.  1.0.18 is the stable version in which it was
> fixed.

1.0.18 is also the version that's in the PPA - 1.08 has to be a typo as the
PPA tracks currently releases closely, and 1.0.8 is 18 months old.

I've seen a similar issue reported with apt-xapian-index in Ubuntu (it uses
Xapian to maintain a database of packages).  But I've never seen anything
like this myself, despite running Ubuntu on my laptop and spending a lot of
my time building Xapian databases.

Xapian's commit operation currently writes data and then calls fdatasync(),
on several files one after another.  That sounds a lot like a bad case in
one of the mails you linked to.

Can you try this patch (you'll need to rebuild Xapian from source, and
depending where you install it, perhaps set LD_LIBRARY_PATH to ensure the new
build gets used):

http://oligarchy.co.uk/xapian/patches/xapian-1.0.18-flint-group-fsyncs.patch

What this does it to at least pair up the calls to fdatasync().  It's
possible to move them all together, but requires more effort, so it'd be
nice to know if this is actually going to help.

Cheers,
    Olly