To: notmuch@notmuchmail.org
From: Olly Betts <olly@survex.com>
Date: Fri, 15 Jan 2010 03:58:50 +0000 (UTC)
Lines: 32
Message-ID: <slrnhkvpvr.2rf.olly@msgid.survex.com>
References: <5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com>
User-Agent: slrn/pre1.0.0-11 (Linux)
Sender: news <news@ger.gmane.org>
Subject: Re: [notmuch] Notmuch performance problems on OSX
Precedence: list

On 2010-01-14, Oliver Charles wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:

One difference between OS X and other systems is that OS X supports the
F_FULLSYNC ioctl, and other systems don't (currently, at least AFAIK)
and Xapian uses that if it is available to ensure that changes have
actually made it to disk:

http://trac.xapian.org/ticket/288

On other systems, it uses fdatasync() or fsync(), which typically just
ensure that the data has left the OS - it can sit in disk controller or
drive caches for potentially seconds longer.  This call happens once
per table for every (explicit or implicit) flush on a database.

I can see an issue here which is that currently Xapian writes the base
file for the table, then syncs it, then does the next table.  I bet it
would be more efficient to write them all and then sync them all,
especially with F_FULLSYNC.

I'll take a look at doing that, and have created a ticket for it:

http://trac.xapian.org/ticket/426

If after that this is still causing problems, it should probably be made
configurable what (if any) flushing is done.  If you're on a UPS-backed
server, you probably don't need such paranoia.

Cheers,
    Olly