Date: Wed, 26 Jan 2011 10:07:28 -0500
From: Austin Clements <amdragon@MIT.EDU>
To: Carl Worth <cworth@cworth.org>
Subject: Re: [PATCH 1/3] new: Do not defer maildir flag synchronization
	during the first run
Message-ID: <20110126150728.GT13226@mit.edu>
References: <1295603977-14326-1-git-send-email-sojkam1@fel.cvut.cz>
	<1295603977-14326-3-git-send-email-sojkam1@fel.cvut.cz>
	<AANLkTinsC+89yx7jGUaB8PLiftLjyTr7gosUDgAC_g0S@mail.gmail.com>
	<87pqrk10wm.fsf@yoom.home.cworth.org>
	<87mxmn27w6.fsf@yoom.home.cworth.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87mxmn27w6.fsf@yoom.home.cworth.org>
User-Agent: Mutt/1.5.20 (2009-06-14)
Cc: notmuch@notmuchmail.org
Precedence: list

Quoth Carl Worth on Jan 26 at  9:59 pm:
> On Wed, 26 Jan 2011 19:15:21 +1000, Carl Worth <cworth@cworth.org> wrote:
> > But then I'm also wondering if perhaps we could do the processing undeferred
> > in all cases?
> > 
> > I haven't thought that through, but I'd be glad to hear your ideas.
> 
> This is still an open question if anyone wants to pursue it, (it might
> make it simpler to then fix the atomicity bugs with adding new messages
> to the database, and only later adjusting the maildir filename).

I believe you're right that synchronization could always be done
eagerly.  In fact, I believe this is true even with the addition of
conjunctive and disjunctive flags.

When adding or removing messages, flag synchronization fully dictates
all tags in flag2tag (that is, the tags' current states don't matter).
It also always examines *all* file names backing the message.  This is
a stateless process.  With eager synchronization, the tags may go
through some transient states during notmuch new, but will always
settle down to the correct set as of the final add or remove for a
given message ID.

> On that topic, if we decide we do need to defer the tags/flags mapping,
> then the real fix is to probably also defer the addition of the filename
> to the message document in the database. If we change these things only
> at the same time, then we should be able to avoid any problems with
> things getting out of synchronization.

I'd been thinking about this as a way to break up notmuch new into
small, consistent transactions, but I don't think it's necessary since
flags can be synchronized eagerly.  In fact, with eager
synchronization and one other change, I believe notmuch new can be
completely correctly performed in lots of small transactions.

The one other change is that currently, if notmuch is interrupted
after updating a directory mtime but before processing file removals,
a subsequent notmuch new will miss those removals.  This could be
brushed under the rug, since notmuch will notice the removal after any
other change in that directory.  Or it could be handled correctly by
giving directories a "removal mtime" in addition to their current
"addition mtime" that is only updated after removals are handled.