Date: Tue, 25 Jan 2011 10:08:12 +1030
From: Tim Stoakes <notmuch@stoakes.net>
To: David Mazieres expires 2011-02-24 PST
	<mazieres-468jumwyp2ei6jwxn8m5vrkyja@temporary-address.scs.stanford.edu>
Subject: Re: Tag timestamps and synchronization
Message-ID: <20110124233812.GB23628@mail.stoakes.net>
References: <x78vyanmnb.wl@ta.scs.stanford.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <x78vyanmnb.wl@ta.scs.stanford.edu>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: notmuch@notmuchmail.org
Precedence: list

dm-list-email-notmuch@scs.stanford.edu(dm-list-email-notmuch@scs.stanford.edu)@240111-11:10:
> One of the features I would like to see from notmuch is an easier
> ability to synchronize tags across machines.  At the very least, I
> would need either incremental dump and restore, or some way to
> communicate arbitrary tags to a local imap server that shares
> notmuch's maildir (much as notmuch currently syncs the standard tags),
> so that I synchronize two maildirs with a tool like offlineimap.

David,

I do something like this by using some shell scripts with formail, to
'store' notmuch tags into the X-Label headers of the individual mails.
Offlineimap then syncs these headers. If I need the tags to become
notmuch-ified on the target, I just scan all the mail's X-Label headers.

(Actually it's better than this, since I use maildrop to set notmuch
tags with notmuch-deliver, *and* set X-Label headers to the same thing,
at mail delivery time. Then I use keybindings and shell scripts in mutt
such that whenever I retag a message, it is pushed to both notmuch and
X-Label.)

I'm happy to share this hack glue if it would help.

This is not great for a few reasons - there are a ton of moving parts,
and some double-work. If notmuch could index X-Label headers (a coming
feature I hear) then this would be much cleaner.

This is just one way of doing it, that works for me...

Tim

> As Carl pointed out to me in private email, there has been some
> previous discussion in the following thread:
> 
>  	notmuch show id:87hbfnmiux.fsf@yoom.home.cworth.org
> 
> Based on that thread, there seems to be some desire for notmuch to
> keep track of a per-message timestamp when the flags were last
> updated.  This would allow much easier expiration for people who want
> the deleted tag.  It would also allow incremental dump and restore of
> tags, which is exactly what I need to sync tags across servers with
> reasonable amounts of bandwidth.
> 
> Metadata timestamps are one of those things that probably have a lot
> of different applications, so since Carl is considering a new database
> format for the next release anyway, perhaps it also makes sense to add
> a metadata change time for each messages.
> 
> The timestamp would be included in "dump" output, and you could
> request a dump of changes since a particular time.  On restore, you
> might have several options:
> 
>   - overwrite: always set the new tags and timestamp in the database
>     to the value in the restore data.
> 
>   - update: always set the tags, but update the to the current time.
> 
>   - conditional T: update only if the message metadata has not been
>     updated since time T.
> 
> To sync flags, then you just need to keep track of the last time you
> synced with a particular server--call this time T.  Do a dump since
> time T, upload to server, do a conditional restore for time T on
> server.  Finally do a partial dump from time T on the server and an
> overwrite import on the client.  (This policy makes changes on the
> server always override conflicting ones on the client--perhaps people
> want other policies, like union of the tags, etc.)
> 
> 
> Second, there seems to be some desire in that thread to sync with IMAP
> flags.  This would be particularly great, but the easies way to do it
> is probably *not* to try to implement IMAP, but rather to use an
> existing IMAP server and just modify the maildir so that the IMAP
> server will pick up the flags.
> 
> In the case of dovecot, the arbitrary tag format is very simple.  Each
> maildir has a file called dovecot-keywords mapping numbers 0, 1,
> ... to keywords.  Then mail file names contain lower-case letters for
> the flags they are marked with--0 => a, 1 => b, etc.--allowing up to
> 26 arbitrary tags for each maildir.  One could probably sync to
> dovecot's maildir format relatively easily in a script given
> incremental dump and restore of tags.  Or possibly notmuch could
> natively support dovecot as one of multiple back-end tag storage
> schemes.
> 
> Having a static tag mapping in the .notmuch-config file would be much
> better than hard-coding flag2tag.  However, I'm not sure it's
> sufficient.  The reason is that if you ever completely delete a tag
> (e.g., you have "todo", and "meeting" tags and periodically have no
> messages in either categories in a given mail folder), then an IMAP
> server like dovecot might end up re-allocating the letters
> corresponding to those tags in a different order.  Also, at least for
> dovecot, the flag mappings are per-folder, which you kind of want
> since you are limited to 26 non-standard tags, so global values might
> not work.
> 
> I'm curious to hear people's thoughts/reactions?
> 
> David

-- 
Tim Stoakes