Re: Deduplication ?

diff --git a/3c/c26265e3f53c1426aad77f5c2d25fbef87b222 b/3c/c26265e3f53c1426aad77f5c2d25fbef87b222
new file mode 100644 (file)
index 0000000..5b6a6d6
--- /dev/null
+++ b/3c/c26265e3f53c1426aad77f5c2d25fbef87b222
@@ -0,0 +1,93 @@
+Return-Path: <david.edmondson@oracle.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 1145A431FAF\r
+       for <notmuch@notmuchmail.org>; Mon,  2 Jun 2014 10:26:10 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.299\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3, UNPARSEABLE_RELAY=0.001]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id RU0FL3VeG2sL for <notmuch@notmuchmail.org>;\r
+       Mon,  2 Jun 2014 10:26:04 -0700 (PDT)\r
+X-Greylist: delayed 13336 seconds by postgrey-1.32 at olra;\r
+       Mon, 02 Jun 2014 10:26:04 PDT\r
+Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 90D4E431FAE\r
+       for <notmuch@notmuchmail.org>; Mon,  2 Jun 2014 10:26:04 -0700 (PDT)\r
+Received: from ucsinet22.oracle.com (ucsinet22.oracle.com [156.151.31.94])\r
+       by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with\r
+       ESMTP id s52HPkfJ009837\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);\r
+       Mon, 2 Jun 2014 17:25:47 GMT\r
+Received: from aserz7022.oracle.com (aserz7022.oracle.com [141.146.126.231])\r
+       by ucsinet22.oracle.com (8.14.5+Sun/8.14.5) with ESMTP id\r
+       s52HPi16022038\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);\r
+       Mon, 2 Jun 2014 17:25:45 GMT\r
+Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20])\r
+       by aserz7022.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id\r
+       s52HPiWt010428; Mon, 2 Jun 2014 17:25:44 GMT\r
+Received: from localhost (/81.149.164.25)\r
+       by default (Oracle Beehive Gateway v4.0)\r
+       with ESMTP ; Mon, 02 Jun 2014 10:25:44 -0700\r
+To: Jani Nikula <jani@nikula.org>, Mark Walters <markwalters1009@gmail.com>,\r
+       Tomi Ollila <tomi.ollila@iki.fi>,\r
+       Vladimir Marek <Vladimir.Marek@oracle.com>, notmuch@notmuchmail.org\r
+Subject: Re: Deduplication ?\r
+In-Reply-To: <87y4xfz1fi.fsf@nikula.org>\r
+References: <20140602123212.GA12639@virt.cz.oracle.com>\r
+       <87d2ers9mi.fsf@qmul.ac.uk> <m2ppirs8ea.fsf@guru.guru-group.fi>\r
+       <87ppirqtfa.fsf@qmul.ac.uk> <87y4xfz1fi.fsf@nikula.org>\r
+User-Agent: Notmuch/0.18 (http://notmuchmail.org) Emacs/24.3.1\r
+       (x86_64-pc-linux-gnu)\r
+Sender: david.edmondson@oracle.com\r
+From: David Edmondson <david.edmondson@oracle.com>\r
+Date: Mon, 02 Jun 2014 18:25:42 +0100\r
+Message-ID: <cunegz71aw9.fsf@gargravarr.hh.sledj.net>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-Source-IP: ucsinet22.oracle.com [156.151.31.94]\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 02 Jun 2014 17:26:10 -0000\r
+\r
+On Mon, Jun 02 2014, Jani Nikula wrote:\r
+>>> One should also have some message content heuristics to determine that the\r
+>>> content is indeed duplicate and not something totally different (not that\r
+>>> we can see the different content anyway... but...)\r
+>>\r
+>> That would be nice.\r
+>\r
+> And quite hard.\r
+\r
+Thinking about this a bit...\r
+\r
+The headers are likely to be different, so you could remove them (get\r
+rid of everything up to the first empty line).\r
+\r
+Various mailing lists add footers, so you would need to remove them (a\r
+regular expression based approach would catch most of them easily).\r
+\r
+The remaining content should be the same for identical messages, so a\r
+sensible hash (md5) could be used to compare.\r
+\r
+Although, some MTAs modify the body of the message when manipulating\r
+encoding. I don't know how to address this.\r