Deduplication ?

diff --git a/0f/c9dac1c24a2d1c2173a8d681e5bbdebdd77c4e b/0f/c9dac1c24a2d1c2173a8d681e5bbdebdd77c4e
new file mode 100644 (file)
index 0000000..f8d26f7
--- /dev/null
+++ b/0f/c9dac1c24a2d1c2173a8d681e5bbdebdd77c4e
@@ -0,0 +1,96 @@
+Return-Path: <Vladimir.Marek@oracle.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 61266431FBC\r
+       for <notmuch@notmuchmail.org>; Mon,  2 Jun 2014 06:22:48 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.299\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3, UNPARSEABLE_RELAY=0.001]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id XIyWT63Mg4Er for <notmuch@notmuchmail.org>;\r
+       Mon,  2 Jun 2014 06:22:42 -0700 (PDT)\r
+X-Greylist: delayed 3017 seconds by postgrey-1.32 at olra;\r
+       Mon, 02 Jun 2014 06:22:41 PDT\r
+Received: from aserp1050.oracle.com (aserp1050.oracle.com [141.146.126.70])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 00227431FAE\r
+       for <notmuch@notmuchmail.org>; Mon,  2 Jun 2014 06:22:41 -0700 (PDT)\r
+Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69])\r
+       by aserp1050.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with\r
+       ESMTP id s52CWOLT026192\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)\r
+       for <notmuch@notmuchmail.org>; Mon, 2 Jun 2014 12:32:24 GMT\r
+Received: from ucsinet21.oracle.com (ucsinet21.oracle.com [156.151.31.93])\r
+       by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with\r
+       ESMTP id s52CWKMI005824\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)\r
+       for <notmuch@notmuchmail.org>; Mon, 2 Jun 2014 12:32:21 GMT\r
+Received: from userz7022.oracle.com (userz7022.oracle.com [156.151.31.86])\r
+       by ucsinet21.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id\r
+       s52CWJ8M027689\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)\r
+       for <notmuch@notmuchmail.org>; Mon, 2 Jun 2014 12:32:20 GMT\r
+Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13])\r
+       by userz7022.oracle.com (8.14.5+Sun/8.14.4) with ESMTP id\r
+       s52CWHIY025494\r
+       for <notmuch@notmuchmail.org>; Mon, 2 Jun 2014 12:32:18 GMT\r
+Received: from virt.cz.oracle.com (/10.163.102.127)\r
+       by default (Oracle Beehive Gateway v4.0)\r
+       with ESMTP ; Mon, 02 Jun 2014 05:32:16 -0700\r
+Date: Mon, 2 Jun 2014 14:32:12 +0200\r
+From: Vladimir Marek <Vladimir.Marek@oracle.com>\r
+To: notmuch@notmuchmail.org\r
+Subject: Deduplication ?\r
+Message-ID: <20140602123212.GA12639@virt.cz.oracle.com>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Disposition: inline\r
+User-Agent: Mutt/1.5.22.1-rc1 (2013-10-16)\r
+X-Source-IP: aserp1040.oracle.com [141.146.126.69]\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 02 Jun 2014 13:22:48 -0000\r
+\r
+Hi,\r
+\r
+I want to import bigger chunk of archived messages into my notmuch\r
+database. It's about 100k messages. The problem is, that I most probably\r
+have quite a lot of those messages in the DB. Basically I would like to\r
+add only those I don't have already.\r
+\r
+There are two possibilities\r
+\r
+a) I will add all the 100k messages and then remove the duplicities.\r
+\r
+b) I will write a script which will parse the message ID's of the\r
+   to-be-added messages and try to match them to the notmuch DB. Adding\r
+   only files I can't find already.\r
+\r
+Ad b) might be better option, but I started to play with the idea of\r
+deduplication. I'm thinking about listing all the message IDs stored in\r
+DB, listing all files belonging to the IDs and deleting all but one.\r
+Also I'm thinking about implementing some simple algorithm telling me\r
+whether the messages are really very similar. Just to be sure I don't\r
+delete something I don't want to.\r
+\r
+Was anyone playing with the idea?\r
+\r
+-- \r
+       Vlad\r