--- /dev/null
+Return-Path: <eirik@eirikba.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 20B74431FAF\r
+ for <notmuch@notmuchmail.org>; Sun, 4 Nov 2012 02:06:22 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id nrU++yEQ6QzM for <notmuch@notmuchmail.org>;\r
+ Sun, 4 Nov 2012 02:06:20 -0800 (PST)\r
+Received: from atmail.labs2.com (atmail.labs2.com [93.182.166.49])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 0B655431FAE\r
+ for <notmuch@notmuchmail.org>; Sun, 4 Nov 2012 02:06:20 -0800 (PST)\r
+Received: from [178.74.1.248] (helo=star.eba)\r
+ by atmail.labs2.com with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77)\r
+ (envelope-from <eirik@eirikba.org>) id 1TUx5q-0006GM-QA\r
+ for notmuch@notmuchmail.org; Sun, 04 Nov 2012 11:06:14 +0100\r
+Received: from eirik by star.eba with local (Exim 4.80)\r
+ (envelope-from <eirik@eirikba.org>) id 1TUx5u-00015r-34\r
+ for notmuch@notmuchmail.org; Sun, 04 Nov 2012 11:06:18 +0100\r
+From: Eirik Byrkjeflot Anonsen <eirik@eirikba.org>\r
+To: notmuch@notmuchmail.org\r
+Subject: Re: Automatic suppression of non-duplicate messages\r
+References: <87mwyz3s9d.fsf@star.eba> <87390qxvb4.fsf@maritornes.cs.unb.ca>\r
+Date: Sun, 04 Nov 2012 11:06:18 +0100\r
+In-Reply-To: <87390qxvb4.fsf@maritornes.cs.unb.ca> (David Bremner's message of\r
+ "Sat, 03 Nov 2012 16:53:19 -0400")\r
+Message-ID: <87wqy1u1gl.fsf@star.eba>\r
+User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux)\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+X-ACL-Warn: Authenticated as: Sent as: eirik@eirikba.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 04 Nov 2012 10:06:22 -0000\r
+\r
+David Bremner <david@tethera.net> writes:\r
+\r
+> Eirik Byrkjeflot Anonsen <eirik@eirikba.org> writes:\r
+>\r
+>> That's not what I see. If I search for a term that only appears in\r
+>> one of the "copies", none of the copies are included in the search\r
+>> result.\r
+>\r
+> The offending code is at line 1813 of lib/database.cc; the message is\r
+> only indexed if the message-id is new.\r
+>\r
+> It might be sensible to move _notmuch_message_index_file into the other\r
+> branch of the if, but even if that works fine, something more\r
+> sophisticated is needed for the call to\r
+> __notmuch_message_set_header_values; the invariant that each message has\r
+> a single subject seems reasonable.\r
+\r
+Hmm, depends. Assuming indexing is intended to be used for searching,\r
+one might want to search for something that occurs in one subject but\r
+not the other. In practice I doubt it matters.\r
+\r
+\r
+> Offhand I'm not sure of a good method of automatically deciding what is\r
+> the same message (with e.g. headers and footer text added by a mailing\r
+> list).\r
+\r
+I don't think the real problem here is the duplicate detection algorithm\r
+itself. It is rather that notmuch forces a particular duplicate\r
+detection algorithm on its users. Duplicate detection should really be\r
+delegated to a different application, thus allowing people to experiment\r
+with whatever algorithm works best for them. (Just like notmuch\r
+delegates the choice of initial tags on messages to an external\r
+application.)\r
+\r
+But first notmuch must be modified so it can sensibly treat multiple\r
+instances having the same message-id as separate messages. That seems\r
+to me to be the hard part. (And some way for external applications to\r
+join and split copies, of course.)\r
+\r
+\r
+\r
+\r
+However, if you want an algorithm that is likely to get rid of most\r
+duplicates while keeping most non-duplicates separate, here's a quick\r
+suggestion:\r
+\r
+\r
+Just to clarify: The goal is to suppress most copies of the same message\r
+while not suppressing a single instance of a different message. It\r
+isn't important if a few duplicate messages makes it through, but it is\r
+imperative that no "real" message is dropped.\r
+\r
+To check whether two instances are duplicates, I suspect something like\r
+this algorithm would be "good enough":\r
+\r
+- Message-Id must be the same. This isn't actually necessary, but it\r
+ makes sense to require it anyway.\r
+\r
+- From and Date must be the same. These form important context that may\r
+ change the meaning of the message (e.g. "me too" depends heavily on\r
+ From, and "let's meet tomorrow" depends heavily on Date). (Are there\r
+ more context-supplying headers we should worry about?)\r
+\r
+- If Subject and body are also the same, the instances are duplicates.\r
+\r
+- Otherwise, if neither of the messages come from a mailing list,\r
+ they're probably not duplicates.\r
+\r
+- Otherwise, grab a few other (recent) mails from the same mailing list.\r
+ If all the bodies end with the same text, ignore that text when\r
+ comparing the bodies.\r
+\r
+- For the Subject, again use a few other (recent) mails from the same\r
+ mailing list for comparison. But this time only look for one of the\r
+ well-known common patterns. If all the mails matches the same\r
+ pattern, ignore that pattern when comparing the Subject.\r
+\r
+- For both of the above, it would be good to pick messages from\r
+ different threads, to avoid accidental similarities. I suspect this\r
+ is more important for subjects than bodies, though.\r
+\r
+- Also, leading and trailing whitespace should probably be dropped.\r
+\r
+- (Some other transformations may make sense, such as reflowing text or\r
+ converting between character sets. In practice I doubt that will make\r
+ much of a difference.)\r
+\r
+- If the "canonicalized" body and Subject are the same, the messages are\r
+ duplicates. At least there's now pretty much no chance that there is\r
+ anything interesting that will be missed by dropping one of the\r
+ messages.\r
+\r
+\r
+(I'm assuming that identifying mailing lists are usually\r
+straightforward, e.g. using the List-Id header).\r
+\r
+eirik\r
projects / notmuch-archives.git / commitdiff