--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id C475B431FC0\r
+ for <notmuch@notmuchmail.org>; Mon, 21 Apr 2014 09:21:10 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 0zhyed3mCmm0 for <notmuch@notmuchmail.org>;\r
+ Mon, 21 Apr 2014 09:21:06 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-7.mit.edu (dmz-mailsec-scanner-7.mit.edu\r
+ [18.7.68.36])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id AFF99431FBC\r
+ for <notmuch@notmuchmail.org>; Mon, 21 Apr 2014 09:21:05 -0700 (PDT)\r
+X-AuditID: 12074424-f79e26d000000c70-48-53554570e296\r
+Received: from mailhub-auth-1.mit.edu ( [18.9.21.35])\r
+ (using TLS with cipher AES256-SHA (256/256 bits))\r
+ (Client did not present a certificate)\r
+ by dmz-mailsec-scanner-7.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 24.4D.03184.07545535; Mon, 21 Apr 2014 12:21:04 -0400 (EDT)\r
+Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
+ by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id s3LGL2ds002667; \r
+ Mon, 21 Apr 2014 12:21:03 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s3LGKxkl020572\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+ Mon, 21 Apr 2014 12:21:01 -0400\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1WcGxn-0006vR-H6; Mon, 21 Apr 2014 12:20:59 -0400\r
+Date: Mon, 21 Apr 2014 12:20:58 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Mark Walters <markwalters1009@gmail.com>\r
+Subject: Re: [RFC PATCH] Re: excessive thread fusing\r
+Message-ID: <20140421162058.GE25817@mit.edu>\r
+References: <87ioq5mrbz.fsf@maritornes.cs.unb.ca> <87fvl8mpzj.fsf@qmul.ac.uk>\r
+ <87oazwjq1e.fsf@yoom.home.cworth.org> <877g6kmcmh.fsf@qmul.ac.uk>\r
+ <8738h7kv2q.fsf@qmul.ac.uk>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=iso-8859-1\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: 8bit\r
+In-Reply-To: <8738h7kv2q.fsf@qmul.ac.uk>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFlrHKsWRmVeSWpSXmKPExsUixCmqrFvgGhpssPGFrsXNn3PYLG60djNa\r
+ rJ7LY3H95kxmBxaP3ZsfsHjsnHWX3ePZqlvMHlsOvWcOYInisklJzcksSy3St0vgyjh5byNT\r
+ wRSNinXTfzM2MP6S72Lk4JAQMJE4tjy9i5ETyBSTuHBvPVsXIxeHkMBsJokTj+4zQTgbGSWu\r
+ PdkP5Zxmkmg4uowZwlkClJl/nw2kn0VAVaKp+RUTiM0moCGxbf9yRhBbREBH4vahBewg65gF\r
+ CiVOnawECQsDbb689AcziM0LVPJ/yzeomZsZJfrbNrFAJAQlTs58AmYzAxXt3HqHDWKOtMTy\r
+ fxwQYXmJ5q2zweZwAq29v+o02DmiAioSU05uY5vAKDwLyaRZSCbNQpg0C8mkBYwsqxhlU3Kr\r
+ dHMTM3OKU5N1i5MT8/JSi3TN9XIzS/RSU0o3MYKihd1FZQdj8yGlQ4wCHIxKPLwSBqHBQqyJ\r
+ ZcWVuYcYJTmYlER5fSyBQnxJ+SmVGYnFGfFFpTmpxYcYJTiYlUR412sC5XhTEiurUovyYVLS\r
+ HCxK4rxvra2ChQTSE0tSs1NTC1KLYLIyHBxKErwvXIAaBYtS01Mr0jJzShDSTBycIMN5gIY/\r
+ AqnhLS5IzC3OTIfIn2JUlBLnVQBJCIAkMkrz4HphyewVozjQK8K8p0CqeICJEK77FdBgJqDB\r
+ T7aEgAwuSURISTUwmieHX6k00/ioMPO9evac4y4/p7WXTdM9/4qvp1CiWEp49751gaY/J7jz\r
+ 7zNg02LkqVl6yHRzf8GL42+9PnEeUvzN2aTfo3bnMkfp/0mHr4gtY1NOuHT3fqJUXz5ruYcV\r
+ e/MDpytzBcVE6lefSHcK8tzwpSxk5sLgRxLb38RxhzlLmQqVLfqkxFKckWioxVxUnAgApCb5\r
+ MEEDAAA=\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 21 Apr 2014 16:21:10 -0000\r
+\r
+Quoth Mark Walters on Apr 21 at 8:20 am:\r
+> \r
+> >> I haven't tracked through all the logic of the existing algorithm for\r
+> >> this case. But I don't like hearing that notmuch constructs different\r
+> >> threads for the same messages presented in different orders. This sounds\r
+> >> like a bug separate from what we've discussed above. \r
+> \r
+> I think I have now found this bug and it is separate from the malformed\r
+> In-Reply-To problems.\r
+> \r
+> The problem is that when we merge threads we update all the thread-ids\r
+> of documents in the loser thread. But we don't (if I understand the code\r
+> correctly) update dangling "metadata" references to threads which don't\r
+> (yet) have any documents.\r
+\r
+This exactly the problem I wrote\r
+id:1395608456-9673-1-git-send-email-amdragon@mit.edu to test, but I\r
+had convinced myself everything was okay because we link a message to\r
+both its parents and all of its children. But that's only true if you\r
+eventually receive the linking message (which in the test I made, you\r
+do). In this case, you never receive the linking message, so even\r
+though notmuch has enough information to bring the two threads\r
+together, it doesn't.\r
+\r
+Maybe I should create a second variant of that test where all of the\r
+messages reference their entire heritage (rather than just their\r
+immediate parent) and test that they're *always* in one thread\r
+regardless of receipt order (rather than only checking once they've\r
+all been received)? I think that would weed out this case.\r
+\r
+> To make this explicit consider the 2 messages 17,18 in the set. \r
+> \r
+> Message 17 has id <87wrkidfrh.fsf@pinto.chemeng.ucl.ac.uk> and has no\r
+> references/in-reply-to so has no parents.\r
+> \r
+> Message 18 has a reference to <87wrkidfrh.fsf@pinto.chemeng.ucl.ac.uk>\r
+> and an in-reply-to to <e.fraga@ucl.ac.uk> and\r
+> <87wrkidfrh.fsf@pinto.chemeng.ucl.ac.uk>\r
+> \r
+> If you add 17 first then it gets thread-id 1 and then when you add 18 message 18 gets\r
+> thread-id 2 as does the dangling link to the "unseen" message\r
+> e.fraga@ucl.ac.uk, and then message 17 is moved to thread-id 2.\r
+> \r
+> However, if you add 18 first then it gets thread-id 1 and the dangling\r
+> link gets id 1. When 17 is added it gets thread-id 2, message 18 gets\r
+> thread-id updated to 2 *but* the dangling link to e.fraga@ucl.ac.uk does\r
+> not get updated so stays thread-id 1.\r
+> \r
+> In particular when 52 comes along with its link to e.fraga@ucl.ac.uk\r
+> then:\r
+> \r
+> In the first case it gets put in thread-id 3 and the other two\r
+> messages get moved into thread 3.\r
+> \r
+> In the second case, message 52 gets put in thread 3 and thread 1\r
+> (the dangling link) gets moved into thread 3 leaving thread 2\r
+> (containing messages 17 and 18) untouched.\r
+\r
+So, there's an obvious, messy way to fix this: update the metadata\r
+references when we do thread renumbering. This is messy because that\r
+data *isn't indexed*. The only way to find the records we need to\r
+update is to scan them all. This isn't completely terrible because\r
+it's a sequential scan and we could cache it in memory, but it\r
+certainly isn't going to help notmuch new's performance. (My database\r
+has 6,749 of these, which takes ~1 second to scan on a cold cache,\r
+though that's with an SSD [1]).\r
+\r
+\r
+But let me propose an idea I've been kicking around for a while: ghost\r
+message documents. Rather than using user metadata for tracking these\r
+missing messages, use regular documents with the exact same terms we\r
+use now for message IDs and thread IDs, but with a Tghost term instead\r
+of a Tmail term to distinguish their type. This solves the problem\r
+using infrastructure we already have in place, simplifies the message\r
+linking code, and may even make it faster. It's a schema update, but\r
+a simple and fast one. I think the hardest part is that things like\r
+notmuch_database_find_message would need to distinguish ghosts and\r
+regular messages (which may require pre-fetching the Tghost or Tmail\r
+posting list to do efficiently).\r
+\r
+This also sets us up to do some cool things in the future, though\r
+they're more invasive. If we have message-like documents for these\r
+ghosts, we can store other message-like metadata as well. If we store\r
+tags on them, then we can keep tags around for deleted messages and\r
+*reapply them* if the message comes back. This would finally fix the\r
+races we have now where, if a message is renamed or moved during a\r
+notmuch new, we may think it's deleted only to reindex it with default\r
+tags on the next run. We could also pre-tag messages that haven't\r
+been indexed yet, say from procmail or when sending a message. This\r
+could simplify or even obviate notmuch insert. If we add message\r
+ctimes as proposed by Dave Mazières, this would give us a place to\r
+store and query ctimes of deleted messages (otherwise it's unclear how\r
+you find out about deletions without a full database scan). In\r
+effect, the database becomes truly monotonic.\r
+\r
+[1] Curious?\r
+ yes n | xapian-inspect postlist.DB | \\r
+ awk '!/Key/ {next} /Key: \\x00\\xc0thread_id_/ {N++} /Key: \\x00\\xd0/ {exit} END {print N}'\r
projects / notmuch-archives.git / commitdiff