--- /dev/null
+Return-Path: <jani@nikula.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 116D1431FAF\r
+ for <notmuch@notmuchmail.org>; Mon, 2 Jun 2014 11:29:39 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id d5Hd1BqBWelG for <notmuch@notmuchmail.org>;\r
+ Mon, 2 Jun 2014 11:29:31 -0700 (PDT)\r
+Received: from mail-wg0-f42.google.com (mail-wg0-f42.google.com\r
+ [74.125.82.42]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client\r
+ certificate requested) by olra.theworths.org (Postfix) with ESMTPS id\r
+ CB6A7431FAE for <notmuch@notmuchmail.org>; Mon, 2 Jun 2014 11:29:30 -0700\r
+ (PDT)\r
+Received: by mail-wg0-f42.google.com with SMTP id y10so5465822wgg.25\r
+ for <notmuch@notmuchmail.org>; Mon, 02 Jun 2014 11:29:29 -0700 (PDT)\r
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+ d=1e100.net; s=20130820;\r
+ h=x-gm-message-state:from:to:subject:in-reply-to:references\r
+ :user-agent:date:message-id:mime-version:content-type;\r
+ bh=94KOYEw+4phKzcpXqklJDclTdrYAbM7A50Le9JpUlLo=;\r
+ b=jVdK7IFpDTPGR0wA1mPjxFIcXVvBLd2lM6xtDRmMW5uFjbrHuMjikZX+EkFkX1miVz\r
+ rBqm1XkfJUZpALD0f3Jsjx8kZRfx1KDkY2OsSEM3iAFTZjJYJUGxDOyrJUm9yxFdWIlv\r
+ 4medO1LKEoY6/kMJRArOv87lNGsdrLptq4keljp+HxAXioElOghDPe2Jz2uu6XecILcz\r
+ Lqas6BWIxCpPXFsEyf+HbyQeI4MnKJGM0+n/z7KYkYK+h4fcLzAimwglpscCrXWkzNK6\r
+ yayt1H7kfXaaDZv0q9wVn4/HttbNdas6Nx24lDNfIIEZeH7j8BycYVnQTr+xM9/2i6Xx\r
+ ALOw==\r
+X-Gm-Message-State:\r
+ ALoCoQlXKq7TIgXGmphuTVSBlkAHKo/hWccCbMmzLsBh4XSCj1j5FStrtsY7mzDEIo00PuF4A1Rf\r
+X-Received: by 10.180.73.66 with SMTP id j2mr1723831wiv.36.1401733769370;\r
+ Mon, 02 Jun 2014 11:29:29 -0700 (PDT)\r
+Received: from localhost (dsl-hkibrasgw2-58c36f-91.dhcp.inet.fi.\r
+ [88.195.111.91]) by mx.google.com with ESMTPSA id\r
+ gp6sm34626931wib.12.2014.06.02.11.29.27 for <multiple recipients>\r
+ (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\r
+ Mon, 02 Jun 2014 11:29:28 -0700 (PDT)\r
+From: Jani Nikula <jani@nikula.org>\r
+To: David Edmondson <david.edmondson@oracle.com>,\r
+ Mark Walters <markwalters1009@gmail.com>, Tomi Ollila <tomi.ollila@iki.fi>,\r
+ Vladimir Marek <Vladimir.Marek@oracle.com>, notmuch@notmuchmail.org\r
+Subject: Re: Deduplication ?\r
+In-Reply-To: <cunegz71aw9.fsf@gargravarr.hh.sledj.net>\r
+References: <20140602123212.GA12639@virt.cz.oracle.com>\r
+ <87d2ers9mi.fsf@qmul.ac.uk> <m2ppirs8ea.fsf@guru.guru-group.fi>\r
+ <87ppirqtfa.fsf@qmul.ac.uk> <87y4xfz1fi.fsf@nikula.org>\r
+ <cunegz71aw9.fsf@gargravarr.hh.sledj.net>\r
+User-Agent: Notmuch/0.18+24~gfe8cd90 (http://notmuchmail.org) Emacs/24.3.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Mon, 02 Jun 2014 21:29:26 +0300\r
+Message-ID: <87vbsjyxkp.fsf@nikula.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 02 Jun 2014 18:29:39 -0000\r
+\r
+On Mon, 02 Jun 2014, David Edmondson <david.edmondson@oracle.com> wrote:\r
+> On Mon, Jun 02 2014, Jani Nikula wrote:\r
+>>>> One should also have some message content heuristics to determine that the\r
+>>>> content is indeed duplicate and not something totally different (not that\r
+>>>> we can see the different content anyway... but...)\r
+>>>\r
+>>> That would be nice.\r
+>>\r
+>> And quite hard.\r
+>\r
+> Thinking about this a bit...\r
+>\r
+> The headers are likely to be different, so you could remove them (get\r
+> rid of everything up to the first empty line).\r
+>\r
+> Various mailing lists add footers, so you would need to remove them (a\r
+> regular expression based approach would catch most of them easily).\r
+\r
+This may work for text/plain messages, but for mime messages (and I\r
+think text/html too) an extra layer of mime structure is usually\r
+added. The problem becomes matching a subtree of mime structure, and\r
+deciding the non-matching layer is noise that can be ignored. The\r
+mailing list manager adding the extra layer may also decode and\r
+reconstruct the existing parts instead of using them as-is.\r
+\r
+> The remaining content should be the same for identical messages, so a\r
+> sensible hash (md5) could be used to compare.\r
+>\r
+> Although, some MTAs modify the body of the message when manipulating\r
+> encoding. I don't know how to address this.\r
+\r
+Let's assume we can figure it all out and find the duplicates. The\r
+question remains, which one to save and which ones to remove? For list\r
+mail, perhaps you'd like to save the copy you received through the list\r
+so you know it's list mail (and you could search for it using list-id:\r
+header *cough* if we indexed that *cough*). Or perhaps you'd like to\r
+save the copy you received directly because some lists let people have\r
+their addresses filtered from cc: header before distributing.\r
+\r
+More useful would probably be raising some flags if the heuristics\r
+detect messages with the same message-id that are clearly *different*\r
+messages. (Perhaps that's what Tomi was after to begin with?)\r
+\r
+Finally, I personally wouldn't want any duplicates removed; rather I'd\r
+like notmuch to index information across all duplicates, and provide UI\r
+features to see the alternatives if desired.\r
+\r
+BR,\r
+Jani.\r
projects / notmuch-archives.git / commitdiff