From: Mark Anderson Date: Fri, 14 Sep 2012 16:27:23 +0000 (+1800) Subject: Re: Better Gmail handling by not using Notmuch tags X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=aad0fc06c500a474446ac361500a4a378f475592;p=notmuch-archives.git Re: Better Gmail handling by not using Notmuch tags --- diff --git a/dc/2717acb7a99aef7d040dc20b9d2530b962fa82 b/dc/2717acb7a99aef7d040dc20b9d2530b962fa82 new file mode 100644 index 000000000..e64306354 --- /dev/null +++ b/dc/2717acb7a99aef7d040dc20b9d2530b962fa82 @@ -0,0 +1,266 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id D5F71431FAF + for ; Fri, 14 Sep 2012 09:27:38 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id wsI8OPNDsxp2 for ; + Fri, 14 Sep 2012 09:27:36 -0700 (PDT) +Received: from db3outboundpool.messaging.microsoft.com + (db3ehsobe001.messaging.microsoft.com [213.199.154.139]) + (using TLSv1 with cipher AES128-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id B8A8F431FAE + for ; Fri, 14 Sep 2012 09:27:35 -0700 (PDT) +Received: from mail26-db3-R.bigfish.com (10.3.81.250) by + DB3EHSOBE004.bigfish.com (10.3.84.24) with Microsoft SMTP Server id + 14.1.225.23; Fri, 14 Sep 2012 16:27:34 +0000 +Received: from mail26-db3 (localhost [127.0.0.1]) by mail26-db3-R.bigfish.com + (Postfix) with ESMTP id 5FDC9400A9; + Fri, 14 Sep 2012 16:27:34 +0000 (UTC) +X-Forefront-Antispam-Report: CIP:163.181.249.108; KIP:(null); UIP:(null); + IPV:NLI; H:ausb3twp01.amd.com; RD:none; EFVD:NLI +X-SpamScore: -13 +X-BigFish: VPS-13(zzbb2dI98dI154cP9371I1432I4015I1415Id6f1izz1202h1d1ah1d2ahzz8275ch17326ah8275bh8275dhz2dh668h839h944hd24hf0ah1220h1288h12a5h12a9h12bdh1155h) +Received: from mail26-db3 (localhost.localdomain [127.0.0.1]) by mail26-db3 + (MessageSwitch) id 1347640053311330_2169; + Fri, 14 Sep 2012 16:27:33 +0000 (UTC) +Received: from DB3EHSMHS005.bigfish.com (unknown [10.3.81.233]) by + mail26-db3.bigfish.com (Postfix) with ESMTP id 3EF501E0071; + Fri, 14 Sep 2012 16:27:33 +0000 (UTC) +Received: from ausb3twp01.amd.com (163.181.249.108) by + DB3EHSMHS005.bigfish.com (10.3.87.105) with Microsoft SMTP Server id + 14.1.225.23; Fri, 14 Sep 2012 16:27:31 +0000 +X-WSS-ID: 0MACLPT-01-A9J-02 +X-M-MSG: +Received: from sausexedgep01.amd.com (sausexedgep01-ext.amd.com + [163.181.249.72]) (using TLSv1 with cipher AES128-SHA (128/128 + bits)) (No + client certificate requested) by ausb3twp01.amd.com (Axway MailGate + 3.8.1) + with ESMTP id 275DA10280AB; Fri, 14 Sep 2012 11:27:29 -0500 (CDT) +Received: from sausexhtp01.amd.com (163.181.3.165) by sausexedgep01.amd.com + (163.181.36.54) with Microsoft SMTP Server (TLS) id 8.3.192.1; + Fri, 14 Sep 2012 11:41:23 -0500 +Received: from adcvmail01.amd.com (163.181.21.78) by sausexhtp01.amd.com + (163.181.3.165) with Microsoft SMTP Server (TLS) id 8.3.213.0; + Fri, 14 Sep 2012 11:27:29 -0500 +Received: from brewmaster.amd.com (brewmaster.amd.com [165.204.144.45]) by + adcvmail01.amd.com (8.13.8/8.13.8) with ESMTP id q8EGRP6i011196; + Fri, 14 Sep 2012 11:27:25 -0500 +Received: from testarossa.amd.com (testarossa.amd.com [165.204.147.44]) by + brewmaster.amd.com (8.13.8/8.13.8) with ESMTP id q8EGRSYK005287; + Fri, 14 Sep 2012 10:27:28 -0600 +Received: (from manderso@localhost) by testarossa.amd.com + (8.13.8/8.13.8/Submit) id q8EGRND8031091; + Fri, 14 Sep 2012 10:27:23 -0600 +X-Authentication-Warning: testarossa.amd.com: manderso set sender to + MarkR.Anderson@amd.com using -f +From: Mark Anderson +To: Rainer M Krug , +Subject: Re: Better Gmail handling by not using Notmuch tags +In-Reply-To: <5052E1A9.8060503@gmail.com> +References: + + + + <5052E1A9.8060503@gmail.com> +User-Agent: Notmuch/0.11.1+251~g1093c24 (http://notmuchmail.org) Emacs/23.2.1 + (i686-pc-linux-gnu) +Date: Fri, 14 Sep 2012 10:27:23 -0600 +Message-ID: <3wda9ws4ljo.fsf@testarossa.amd.com> +MIME-Version: 1.0 +Content-Type: text/plain; charset="us-ascii" +X-OriginatorOrg: amd.com +Cc: Jeremy Nickurak +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Fri, 14 Sep 2012 16:27:39 -0000 + +On Fri, 14 Sep 2012 09:50:01 +0200, Rainer M Krug wrote: +> -----BEGIN PGP SIGNED MESSAGE----- +> Hash: SHA1 +> +> On 13/09/12 17:15, Damien Cassou wrote: +> > On Thu, Sep 13, 2012 at 5:13 PM, Jeremy Nickurak +> > wrote: +> >> Gmail doesn't have folders, of course, it has labels, which are +> >> approximately equivalent to notmuch tags. The key difference being +> >> that a message can only be in one folder, but it can have multiple +> >> tags/labels. +> > +> > Gmail exports its labels as IMAP folders: an email with multiple +> > labels will be duplicated in multiple folders (one per +> > label). That's why I'm asking if it would be possible to manupale +> > folders from Notmuch instead of tags. +> > +> +> I don't think there is an easy solution. notmuch uses a maildir and +> tags. Gmail needs to be synced to this local maildir earlier, and this +> is where I think the problem comes in: I am not aware of any sync tool +> which maintains the gmail labels, as they are in in the imap context +> folders. +> +> I think the only real solution woud be: +> +> Download from gmail -> local: +> 1) download only the "All Mail" folder +> 2) implement a tagging tool which syncs the gmail labels to notmuch tags + +Gmail's IMAP protocol does expose a folder hierarchy which you can use +to reverse engineer the tag cloud of each email. + +Using Offlineimap will happily sync a your Gmail such that each mail +will show up in every folder corresponding to the tags with which the +mail is tagged. When notmuch scans the mail, it will collapse these +multiple mails into one mail message by message-id: and you just need a +script to translate folders into tags. The incoherence between notmuch +tag and mail folders then communicates something different if you are +pre-sync or post-sync. I am calling the offlineimap call "sync", so once +we run it (the first time for sure) we are in 'post-sync'. + + +You might want to take this chance to make your tag cloud coherent +between Notmuch and what exists in the folders, which works out to +something like this for every tag/folder pair in your gmail IMAP +directory (assuming it's synced to a Maildir repository) and notmuch DB. + + notmuch tag +TagX folder:FolderX and not tag:TagX + +Although strictly speaking it isn't actually necessary to put the +"and not tag:TagX" term, as this term this is now implicitly added by +notmuch to improve performance and avoid touching database entries that +already have the tag, I include it to demonstrate the coherence between +TagX and FolderX. + +Then you'll want to handle TagX being removed on the Gmail side too, +so you'll want to do something like this to remove the notmuch TagX. + + notmuch tag -TagX tag:TagX and not folder:FolderX + +Then there's the reverse direction to consider. + +When I see TagX in notmuch, and using FolderX as the proxy for Gmail +tags, then I assume that the user added TagX in notmuch and I need to +synchronize the change. This one is a bit trickier. + + notmuch search --output=files tag:TagX and not folder:FolderX + +will give me the list of filenames, but I need to add them to a folder, +so it's time for bash, or your favorite script language. Spaces in +filenames or tags are your bane here, then you'll want to do something +fancier than just the $() interpolation. + +notmuch search --output=files tag:notmuch and not folder:notmuch | +xargs perl -e'while (defined($_ = shift(@ARGV))) {my $file = +filename($_); system("cp $_ $MAILDIR/notmuch/cur/$file");}' + +Then when I see a mail in FolderX without TagX, that indicates that I +have removed the tag from notmuch, so I will want to remove the copy of +the file from FolderX, and only FolderX. + +This requires you to scan through the list of filenames associated with +the msg-id and delete the file in FolderX. + +Of course this is terrible on performance, as you will have lots of +copies of mails when you have lots of tags on your mail, but here's a +summary of the actions you need to coordinate to keep them in sync. + + +Stage FolderX TagX Action +===== ======= ==== ====== +post-sync No No No Action + +post-sync No Yes Gmail TagX removed caused FolderX copy + to be removed, remove Notmuch TagX + +post-sync Yes No Gmail TagX added, add Notmuch TagX + +post-sync Yes Yes No Action + +pre-sync No No No Action + +pre-sync No Yes Notmuch TagX added, copy mail to FolderX + to add Gmail TagX corresponding to + notmuch TagX + +pre-sync Yes No Notmuch TagX removed, delete mail copy + in FolderX + +pre-sync Yes Yes No Action + +The worst part about this, is that any interrupted action must be +retried until successful, unless we have information about the relative +times of the actions. If instead of trying to rationalize the two +"states" of my messages, I was trying to synchronize the changes, then I +just need to go down the list of changes and take the appropriate action. + +You might have other actions, such as a true delete in Notmuch, which +should remove all copies of the email before doing a mail sync. + +The benefit of using the mail sync is it uses a widely distributed mail +synchronization model, but it really tags expensive to synchronize. It +gets better if you use the Gmail imap extensions that can list the tags +without your client requesting a copy of the entire email for each tag +the mail has. However, Even when you have that, you don't have +bulletproof mail, because the actions need to be guaranteed to complete +before synchronization and after synchronization, and any user changes +need to be held off, as they _will_ be interpreted incorrectly if they +take place during the pre-sync, sync, post-sync window. + +You can simplify this if you make guarantees in your usage model. That +you will never do tagging operations during a pre-, sync, post- cycle, +or that you only do synchronization one way or the other, instead of +full bidirectional sync. + +It's a difficult problem, I look forward to seeing other solutions +proposed. + +Thanks, +-Mark Anderson + +> upload local -> gmail +> 1) upload "All Mail folder +> 2) assign on gmail the labels corresponding to the notmuch tags. +> +> The step 1 could be done by any sync tool available for this (offlineimap, ...) +> +> step 2 needs to be developed - no idea how, but it surely would be really usefull, because then +> notmuch would even become a perfect tool for gmail backup as well. +> +> Cheers, +> +> Rainer +> -----BEGIN PGP SIGNATURE----- +> Version: GnuPG v1.4.11 (GNU/Linux) +> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ +> +> iEYEARECAAYFAlBS4akACgkQoYgNqgF2egoGhwCaAgfXQUAK4RK1v22JOhgYXfR1 +> +C8AnRU892SrxK7IYN9xoxhM865Y+vTA +> =ma75 +> -----END PGP SIGNATURE----- +> +> _______________________________________________ +> notmuch mailing list +> notmuch@notmuchmail.org +> http://notmuchmail.org/mailman/listinfo/notmuch +> +