Re: [PATCH] Add configurable changed tag to messages that have been changed on disk

diff --git a/08/93a19356d6fae63e51dbb9b82673a2a49e9c6f b/08/93a19356d6fae63e51dbb9b82673a2a49e9c6f
new file mode 100644 (file)
index 0000000..274b49e
--- /dev/null
+++ b/08/93a19356d6fae63e51dbb9b82673a2a49e9c6f
@@ -0,0 +1,121 @@
+Return-Path:\r
+ <return-qins2mg6jwx9mkqf94wrbsevww@temporary-address.scs.stanford.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 304B5431FBC\r
+       for <notmuch@notmuchmail.org>; Sun,  6 Apr 2014 13:19:30 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id s62A4tGVgLJO for <notmuch@notmuchmail.org>;\r
+       Sun,  6 Apr 2014 13:19:24 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 26004431FB6\r
+       for <notmuch@notmuchmail.org>; Sun,  6 Apr 2014 13:19:24 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (localhost.scs.stanford.edu\r
+ [127.0.0.1])  by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id\r
+ s36KJLvB015571;       Sun, 6 Apr 2014 13:19:21 -0700 (PDT)\r
+Received: (from dm@localhost)\r
+       by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id s36KJKZv025070;\r
+       Sun, 6 Apr 2014 13:19:20 -0700 (PDT)\r
+X-Authentication-Warning: market.scs.stanford.edu: dm set sender to\r
+       return-qins2mg6jwx9mkqf94wrbsevww@ta.scs.stanford.edu using -f\r
+From: David Mazieres <dm-list-email-notmuch@scs.stanford.edu>\r
+To: Gaute Hope <eg@gaute.vetsj.com>, notmuch@notmuchmail.org\r
+Subject: Re: [PATCH] Add configurable changed tag to messages that have been\r
+       changed on disk\r
+In-Reply-To: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com>\r
+References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com>\r
+Date: Sun, 06 Apr 2014 22:19:19 +0200\r
+Message-ID: <87wqf2gqig.fsf@ta.scs.stanford.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+Reply-To: David Mazieres expires 2014-07-05 CEST\r
+       <mazieres-2vu8n5xbqk4r8zkqh82n4qgci6@temporary-address.scs.stanford.edu>\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 06 Apr 2014 20:19:30 -0000\r
+\r
+Gaute Hope <eg@gaute.vetsj.com> writes:\r
+\r
+> When one of the source files for a message is changed on disk, renamed,\r
+> deleted or a new source file is added. A configurable changed tag is\r
+> is added. The tag can be configured under the option 'changed_tags' in\r
+> the [new] section, the default is none. Tests have been updated to\r
+> accept the new config option.\r
+>\r
+> notmuch-setup now asks for a changed tag after the new tags question.\r
+>\r
+> This could be useful for for example 'afew' to detect remote changes in\r
+> IMAP folders and update the FolderNameFilter to also add tags or remove\r
+> tags when a _existing_ message has been added to or removed from a\r
+> maildir.\r
+\r
+I think this is the wrong way to achieve such functionality, because\r
+then the change tag A) is expensive to remove, B) is easy to misuse\r
+(remember to call fsync everywhere before deleting the change tag), and\r
+C) can be used by only one application.\r
+\r
+A better approach would be to add a new "modtime" xapian value that is\r
+updated whenever the tags or any other terms (such as XFDIRENTRY) are\r
+added to or deleted from a docid.  If it's a Xapian value, rather than a\r
+term, then modtime will be queriable just like date, allowing multiple\r
+applications to query all docids modified since the last time they ran.\r
+\r
+I currently have multiple applications that could significantly benefit\r
+from such a modtime.  An obvious one is proper incremental backups with\r
+notmuch-dump.\r
+\r
+Another example is a tool I have that synchromizes maildirs and notmuch\r
+tags across machines.  With the current interface, there is no way to do\r
+this without scanning the entire database, because any message, even a\r
+very old one, may have changed tags or links.  Moreover, something like\r
+notmuch-dump is way, way too slow to run every time you want to check\r
+for new mail.  notmuch-dump costs 5-10 seconds on my 110,000-message\r
+maildir!  In fact, any approach the gathers tags associated with each\r
+individual docid is a complete non-starter, forcing me to violate\r
+abstraction and examine the postlists associated with each tag and\r
+XFDIRENTRY term.  Even my highly optimized implementation takes about\r
+250 msec (1400 msec on a 32-bit machine), which adds perceptible latency\r
+to synchronizing my clients' notmuch maildirs with my server's when I\r
+poll for new mail.\r
+\r
+Yet another application is something like nottoomuch-addresses, which\r
+currently uses an occasionally incorrect heuristic to detect new\r
+messages based on the Date header.\r
+\r
+Let me make a stronger statement, which is that not only are\r
+modification times an incredibly useful and general primitive, but lack\r
+of modification times is the single thing that kept me away from notmuch\r
+despite years of wanting to switch.  In the end, I invested months\r
+developing a highly-optimized change detector that efficiently diffs\r
+Xapian's Btrees against a mysql database with a snapshot of the same\r
+information.  My solution works, and I now enjoy a replicated notmuch\r
+setup synchronized across three machines, including offline access on my\r
+laptop.  But my 4,000-line C++ program might have been a 400-line shell\r
+script if only notmuch supported docid mod times.\r
+\r
+Also, to put this in perspective, how long does it take to remove the\r
+changed tags from a bunch of messages?  If it's longer than 300 msec on\r
+a 64-bit machine, then even with a single application you'd be better\r
+off using my crazy on-the-side mysql version vector scheme.\r
+\r
+David\r