Re: [PATCH 00/11] Add ghost messages and fix thread linking

diff --git a/f5/3205bfe73f9b8e54379d6e2b218b1ecf8a3ddb b/f5/3205bfe73f9b8e54379d6e2b218b1ecf8a3ddb
new file mode 100644 (file)
index 0000000..06b5783
--- /dev/null
+++ b/f5/3205bfe73f9b8e54379d6e2b218b1ecf8a3ddb
@@ -0,0 +1,103 @@
+Return-Path: <aclements@csail.mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 4EF98431FBC\r
+       for <notmuch@notmuchmail.org>; Sat,  4 Oct 2014 13:15:26 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id NERxwWXStbjr for <notmuch@notmuchmail.org>;\r
+       Sat,  4 Oct 2014 13:15:19 -0700 (PDT)\r
+Received: from outgoing.csail.mit.edu (outgoing.csail.mit.edu [128.30.2.149])\r
+       by olra.theworths.org (Postfix) with ESMTP id 78308431FB6\r
+       for <notmuch@notmuchmail.org>; Sat,  4 Oct 2014 13:15:19 -0700 (PDT)\r
+Received: from [104.131.20.129] (helo=awakeningjr)\r
+       by outgoing.csail.mit.edu with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16)\r
+       (Exim 4.72) (envelope-from <aclements@csail.mit.edu>)\r
+       id 1XaVjW-0006BA-Af; Sat, 04 Oct 2014 16:15:14 -0400\r
+Received: from amthrax by awakeningjr with local (Exim 4.84)\r
+       (envelope-from <aclements@csail.mit.edu>)\r
+       id 1XaVjV-00054i-If; Sat, 04 Oct 2014 16:15:13 -0400\r
+From: Austin Clements <aclements@csail.mit.edu>\r
+To: Tomi Ollila <tomi.ollila@iki.fi>, notmuch@notmuchmail.org\r
+Subject: Re: [PATCH 00/11] Add ghost messages and fix thread linking\r
+In-Reply-To: <m2oatsz16v.fsf@guru.guru-group.fi>\r
+References: <1412345958-8278-1-git-send-email-aclements@csail.mit.edu>\r
+       <m2oatsz16v.fsf@guru.guru-group.fi>\r
+User-Agent: Notmuch/0.18.1+86~gef5e66a (http://notmuchmail.org) Emacs/24.3.1\r
+       (x86_64-pc-linux-gnu)\r
+Date: Sat, 04 Oct 2014 16:15:13 -0400\r
+Message-ID: <87ppe7d226.fsf@csail.mit.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sat, 04 Oct 2014 20:15:26 -0000\r
+\r
+On Sat, 04 Oct 2014, Tomi Ollila <tomi.ollila@iki.fi> wrote:\r
+> On Fri, Oct 03 2014, Austin Clements <aclements@csail.mit.edu> wrote:\r
+>\r
+>> This series modifies our database representation of messages that have\r
+>> been referenced by other messages, but for which we don't have the\r
+>> message itself.  Currently, we store this information as Xapian\r
+>> metadata, but this has several downsides for performance and\r
+>> complexity and results in hard-to-fix thread linking bugs.  This patch\r
+>> series implements "ghost messages", which replace this Xapian metadata\r
+>> with Xapian documents that look and act very much like regular message\r
+>> documents, but simply have no content.  This simplifies and speeds up\r
+>> our thread linking algorithm and fixes the currently broken thread\r
+>> linking test.\r
+>>\r
+>> Ghost messages also open up interesting future possibilities, such as\r
+>> "pre-seeding" tags for messages that are not yet indexed.  This could\r
+>> be used to make notmuch insert simpler and more robust, as part of tag\r
+>> synchronization, and to improve nmbug's behavior when tags arrive\r
+>> before messages.\r
+>\r
+> The code looks OK to me -- there are IMO some strange things but those\r
+> don't change the status quo -- I did look a little past the diffs into\r
+> the code to understand something...\r
+>\r
+> I am now having these patches applied to my 'home' notmuch and haven't yet\r
+> seen anything strange there. In this setup I have 27 emails missing that\r
+> nmbug expects there to be -- let's see if I can get ghost messages there.\r
+\r
+In its current form, it won't help with these missing messages from\r
+nmbug.  In the future, nmbug could use ghost messages to record the tags\r
+for those messages anyway and then it becomes an interface question of\r
+whether nmbug should report such missing messages.\r
+\r
+> ... and tests pass, ran while writing the above part...\r
+>\r
+> I'll put these in use next week on one of my 'work' notmuch. There I have\r
+> seen a problem that while I am on one thread, \r
+> `notmuch-poll-and-refresh-buffer' picks new mail but suddenly this thread\r
+> now has changed it's thread id :O -- making the search buffer go blank.\r
+> I'll see whether it still happens with these (may be totally unrelated)\r
+> and perhaps investigate further...\r
+\r
+I have this same problem with some people I communicate with.  I haven't\r
+looked in to it, but I assume their messages reference only the\r
+immediate parent, so if it's a reply to another unindexed message then\r
+the thread ID may change depending on indexing order and other arbitrary\r
+factors.  I may be wrong, but if this is the case, I don't think ghost\r
+messages will change this behavior.  Solving this may require more\r
+intelligently picking the "winning" thread ID when merging threads;\r
+e.g., pick the thread ID with the most messages, rather than picking\r
+arbitrarily.\r