DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=message-id:date:from:subject:to:in-reply-to:references;
	b=wgW6pmGh2SMm6/hzrJ7mN000FAoFnqKEV4XYmo5yOhA4w0UZl19F0FyYMMc7d89fRe
	iHK7xBhGT0xueagZ9aONNuYbTD8ChFXbwI2Qf0qPf5sDSv4mBQ5xp0Y4iI2nhU6O8ok9
	zE+9+l/LlMFAs/mQj9pW6N4CIpjzP3o29CyA0=
Message-ID: <4b9e6e80.09b6660a.6769.6832@mx.google.com>
Date: Mon, 15 Mar 2010 10:29:36 -0700 (PDT)
From: Ben Gamari <bgamari.foss@gmail.com>
To: Olly Betts <olly@survex.com>, notmuch@notmuchmail.org
In-Reply-To: <slrnhprvfv.hu6.olly@msgid.survex.com>
References: <4b9dccc0.c6c1f10a.3671.44ec@mx.google.com>
	<20100315090401.GA29891@glaive.weftsoar.net>
	<slrnhprvfv.hu6.olly@msgid.survex.com>
Subject: Re: [notmuch] Notmuch performance (literally, in my case)
Precedence: list

On Mon, 15 Mar 2010 09:29:35 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> On 2010-03-15, Hans Dieter Pearcey wrote:
> > On Sun, 14 Mar 2010 22:59:28 -0700 (PDT), Ben Gamari wrote:
> >> Notmuch is using xapian 1.08-1.99karmic from the Xapian backports PPA, which
> >> I believe includes the recent database update optimizations.
> >
> > As far as I know, it doesn't.  1.0.18 is the stable version in which it was
> > fixed.
> 
> 1.0.18 is also the version that's in the PPA - 1.08 has to be a typo as the
> PPA tracks currently releases closely, and 1.0.8 is 18 months old.

Yep, my bad. That was a typo.

> 
> I've seen a similar issue reported with apt-xapian-index in Ubuntu (it uses
> Xapian to maintain a database of packages).  But I've never seen anything
> like this myself, despite running Ubuntu on my laptop and spending a lot of
> my time building Xapian databases.
>
> Can you try this patch (you'll need to rebuild Xapian from source, and
> depending where you install it, perhaps set LD_LIBRARY_PATH to ensure the new
> build gets used):
> 
> http://oligarchy.co.uk/xapian/patches/xapian-1.0.18-flint-group-fsyncs.patch
> 
> What this does it to at least pair up the calls to fdatasync().  It's
> possible to move them all together, but requires more effort, so it'd be
> nice to know if this is actually going to help.
> 

This does seem to help. Of course, latency is a difficult thing to measure, but
notmuch does _feel_ faster. That being said, iostat still only shows
700kByte/second read and 300kByte/second write, so things haven't changed in
the throughput side of things.