Re: [RFC PATCH] Re: excessive thread fusing

diff --git a/32/6e489b2f8098a071a15b42d0b353708ff99b12 b/32/6e489b2f8098a071a15b42d0b353708ff99b12
new file mode 100644 (file)
index 0000000..425bcb7
--- /dev/null
+++ b/32/6e489b2f8098a071a15b42d0b353708ff99b12
@@ -0,0 +1,96 @@
+Return-Path: <david@tethera.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 17791431FBD\r
+       for <notmuch@notmuchmail.org>; Sun, 20 Apr 2014 05:59:57 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id BaCWYoFZAnqA for <notmuch@notmuchmail.org>;\r
+       Sun, 20 Apr 2014 05:59:49 -0700 (PDT)\r
+Received: from mx.xen14.node3324.gplhost.com (gitolite.debian.net\r
+       [87.98.215.224]) (using TLSv1 with cipher AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id EB126431FBC\r
+       for <notmuch@notmuchmail.org>; Sun, 20 Apr 2014 05:59:48 -0700 (PDT)\r
+Received: from remotemail by mx.xen14.node3324.gplhost.com with local (Exim\r
+       4.72) (envelope-from <david@tethera.net>)\r
+       id 1WbrLV-0005Wu-Od; Sun, 20 Apr 2014 12:59:45 +0000\r
+Received: (nullmailer pid 17456 invoked by uid 1000); Sun, 20 Apr 2014\r
+       12:59:26 -0000\r
+From: David Bremner <david@tethera.net>\r
+To: Carl Worth <cworth@cworth.org>, Mark Walters <markwalters1009@gmail.com>, \r
+       notmuch <notmuch@notmuchmail.org>\r
+Subject: Re: [RFC PATCH] Re: excessive thread fusing\r
+In-Reply-To: <87oazwjq1e.fsf@yoom.home.cworth.org>\r
+References: <87ioq5mrbz.fsf@maritornes.cs.unb.ca> <87fvl8mpzj.fsf@qmul.ac.uk>\r
+       <87oazwjq1e.fsf@yoom.home.cworth.org>\r
+User-Agent: Notmuch/0.17+202~gb65f328 (http://notmuchmail.org) Emacs/24.3.1\r
+       (x86_64-pc-linux-gnu)\r
+Date: Sun, 20 Apr 2014 21:59:26 +0900\r
+Message-ID: <87fvl8upg1.fsf@maritornes.cs.unb.ca>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 20 Apr 2014 12:59:57 -0000\r
+\r
+Carl Worth <cworth@cworth.org> writes:\r
+>\r
+> Another idea would be to trigger specifically on common forms. Judging\r
+> From the samples in this particular thread, it seems like a workable\r
+> heuristic would be:\r
+>\r
+>      If the In-Reply-To header begins with '<':\r
+>\r
+>              Parse that initial portion as a message ID\r
+>\r
+>      Else if it ends with '>':\r
+>\r
+>              Parse that final portion as a message ID\r
+>\r
+>      Else\r
+>\r
+>              Ignore this garbage-valued header.\r
+>\r
+\r
+using the hacky script below, I scanned my own mail collection of about\r
+300k messages. I can make the following observations\r
+\r
+- I have some RFC compliant in-reply-to's with multiple ids\r
+- I have have a non-trivial number of Message from $NAME <address> of $date <id>\r
+- I didn't see any cases where using the last angle bracketed thing\r
+  would fail.\r
+- I did see some some cases where the header starts with '<' but the\r
+  matching '>' was missing\r
+- I also noticed some rfc2047 encoding of in-reply-to headers.\r
+\r
+\r
+######################################################################\r
+# hacky script follows\r
+dir=$1\r
+echo Scanning $dir\r
+\r
+tempdir=$(mktemp -d)\r
+echo Writing to ${tempdir}\r
+\r
+find $dir -exec sh -c "formail -c -xIn-reply-to < {}" \; \\r
+  > ${tempdir}/ids\r
+\r
+sed  -e 's/\t/ /' -e 's/   */ /g' -e 's/<[^ ]*>/<id>/g' -e 's/(.*)/(comment)/' < ${tempdir}/ids | sort | uniq | tee ${tempdir}/report\r