From: Ethan Glasser-Camp <ethan.glasser.camp@gmail.com>
To: Austin Clements <amdragon@MIT.EDU>, David Bremner <bremner@debian.org>
Subject: Re: More ideas about logging.
In-Reply-To: <20111216040722.GC12245@mit.edu>
References: <87obv9i7y3.fsf@zancas.localnet> <20111216040722.GC12245@mit.edu>
User-Agent: Notmuch/0.14+45~g6ea9330 (http://notmuchmail.org) Emacs/23.3.1
	(x86_64-pc-linux-gnu)
Date: Fri, 12 Oct 2012 12:28:36 -0400
Message-ID: <87y5jbac3v.fsf@betacantrips.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: Olly Betts <olly@survex.com>, Notmuch Mail <notmuch@notmuchmail.org>
Precedence: list

Austin Clements <amdragon@MIT.EDU> writes:

> The trouble with this approach is that the OS doesn't have to flush
> logfile to the disk platters in any particular order relative to the
> updates to Xapian.  So, after someone trips over your plug, you could
> come back with Xapian saying you have 500 log entries when your
> logfile comes back with only 20.  The only way I know of to fix this
> is to fsync after the logfile write, which would obviously have
> performance issues.  But maybe there are cleverer ways?

Sorry to jump in almost a year after the fact, but..

How bad do you think those performance issues are going to be? I don't
see them as prohibitive, even in the case where you write a log entry
for every message being tagged. Xapian's doing an fsync each time we
commit, isn't it? (Or is there some cute trick where it rename()s the
database?)

If, instead of truncating, you replay logged operations, I think you can
get away with just writing log entries for user-level operations (like
"notmuch tag +mytag to:somequery") which could touch a lot of
messages. This would then only require one fsync on the log file before
doing a lot of tag updates.

Ethan