--- /dev/null
+Return-Path: <jani@nikula.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id AB920431FAF\r
+ for <notmuch@notmuchmail.org>; Fri, 1 Mar 2013 09:06:34 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 1CAHXDA1aJ1Y for <notmuch@notmuchmail.org>;\r
+ Fri, 1 Mar 2013 09:06:31 -0800 (PST)\r
+Received: from mail-la0-f46.google.com (mail-la0-f46.google.com\r
+ [209.85.215.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id B9EFD431FAE\r
+ for <notmuch@notmuchmail.org>; Fri, 1 Mar 2013 09:06:30 -0800 (PST)\r
+Received: by mail-la0-f46.google.com with SMTP id fq12so3176340lab.33\r
+ for <notmuch@notmuchmail.org>; Fri, 01 Mar 2013 09:06:29 -0800 (PST)\r
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+ d=google.com; s=20120113;\r
+ h=x-received:from:to:cc:subject:in-reply-to:references:user-agent\r
+ :date:message-id:mime-version:content-type:x-gm-message-state;\r
+ bh=bq1QZUQOeVGm/fkLALOV81TtqSt+khqkR35uBp40gmA=;\r
+ b=dZ/EjFDPTyTbVj+yqasnGu32v5tkmiUvQ49MQjMeiDIS9zelEwlZQfTQIqd1j5Jl4v\r
+ CTvvRAKyyJUZp5tnG6Ri4Oqw8wEtfNO/+P90wTDhkGVOSje/BujhGC+zuBgzl3txVzE2\r
+ fLfsBlFjAa56H2WQn+YiSP5NMMI1bSRtLBf1WKA+6aYmniNNk9dojPyZxFcfgrMdYluV\r
+ HnBwDh8mi1wNVUnuiL550fix0zZcxofrWu56cW/oe6F5IUXgGXLltyOxCrrXNz3pf9XG\r
+ 6ZEB2xa1qJWM+ohqgWqtqV2n3vzF0GVGneo8ksp34WwagiJ5qEGjIjDeab5i5fKnyy49\r
+ TQEg==\r
+X-Received: by 10.112.36.2 with SMTP id m2mr1005714lbj.100.1362157589105;\r
+ Fri, 01 Mar 2013 09:06:29 -0800 (PST)\r
+Received: from localhost (dsl-hkibrasgw4-50df51-27.dhcp.inet.fi.\r
+ [80.223.81.27])\r
+ by mx.google.com with ESMTPS id b13sm4314722lbd.10.2013.03.01.09.06.27\r
+ (version=TLSv1.2 cipher=RC4-SHA bits=128/128);\r
+ Fri, 01 Mar 2013 09:06:28 -0800 (PST)\r
+From: Jani Nikula <jani@nikula.org>\r
+To: Aaron Ecay <aaronecay@gmail.com>, notmuch@notmuchmail.org\r
+Subject: Re: [RFC] [PATCH] lib/database.cc: change how the parent of a message\r
+ is calculated\r
+In-Reply-To: <87621cteeb.fsf@nikula.org>\r
+References: <1361836225-17279-1-git-send-email-aaronecay@gmail.com>\r
+ <87621cteeb.fsf@nikula.org>\r
+User-Agent: Notmuch/0.15.2+33~g98253a3 (http://notmuchmail.org) Emacs/24.2.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Fri, 01 Mar 2013 19:06:28 +0200\r
+Message-ID: <871ubzt5gr.fsf@nikula.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-Gm-Message-State:\r
+ ALoCoQkhrm09jETl70ZXzygC8E0PKHUI9zJQ0/JA3/vZ22tzJZpsbeqdu5GU6lhFJhSz2WhoWuNB\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 01 Mar 2013 17:06:34 -0000\r
+\r
+On Thu, 28 Feb 2013, Jani Nikula <jani@nikula.org> wrote:\r
+> Hi Aaron -\r
+>\r
+> On Tue, 26 Feb 2013, Aaron Ecay <aaronecay@gmail.com> wrote:\r
+>> Presently, the code which finds the parent of a message as it is being\r
+>> added to the database assumes that the first Message-ID-like substring\r
+>> of the In-Reply-To header is the parent Message ID. Some mail clients,\r
+>> however, put stuff other than the Message-ID of the parent in the\r
+>> In-Reply-To header, such as the email address of the sender of the\r
+>> parent. This can fool notmuch.\r
+>\r
+> I think the background is that RFC 822 defines In-Reply-To (and\r
+> References too for that matter) as *(phrase / msg-id), while RFC 2822\r
+> defines them as 1*msg-id. I'd like something about RFC 822 being\r
+> mentioned in the commit message.\r
+>\r
+> The problem in the gmane message you link to in\r
+> id:87liaa3luc.fsf@gmail.com is likely related to the FAQ item 05.26 "How\r
+> do I fix a bogus In-Reply-To or missing References field?" in the MH FAQ\r
+> http://www.newt.com/faq/mh.html.\r
+>\r
+>> The updated algorithm prefers the last Message ID in the References\r
+>> header. The References header lists messages oldest-first, so the last\r
+>> Message ID is the parent (RFC2822, p. 24). The References header is\r
+>> also less likely to be in a non-standard\r
+>> syntax (http://cr.yp.to/immhf/thread.html,\r
+>> http://www.jwz.org/doc/threading.html). In case the References header\r
+>> is not to be found, fall back to the old behavior.\r
+>> ---\r
+>>\r
+>> I especially notice this problem on public mailing lists, where\r
+>> certain people's messages always cause an "out-dent" of the threading,\r
+>> instead of being nested under whichever message they are replies to.\r
+>>\r
+>> Technically, putting non-Message-ID crud in the In-Reply-To field is a\r
+>> violation of RFC2822, but it appears that in practice the References\r
+>> header is respected more often than the In-Reply-To one.\r
+>>\r
+>> lib/database.cc | 30 ++++++++++++++++++++++--------\r
+>> 1 file changed, 22 insertions(+), 8 deletions(-)\r
+>>\r
+>> diff --git a/lib/database.cc b/lib/database.cc\r
+>> index 91d4329..cbf33ae 100644\r
+>> --- a/lib/database.cc\r
+>> +++ b/lib/database.cc\r
+>> @@ -501,8 +501,10 @@ _parse_message_id (void *ctx, const char *message_id, const char **next)\r
+>> * 'message_id' in the result (to avoid mass confusion when a single\r
+>> * message references itself cyclically---and yes, mail messages are\r
+>> * not infrequent in the wild that do this---don't ask me why).\r
+>> + *\r
+>> + * Return the last reference parsed.\r
+>> */\r
+>> -static void\r
+>> +static char *\r
+>> parse_references (void *ctx,\r
+>> const char *message_id,\r
+>> GHashTable *hash,\r
+>> @@ -511,7 +513,7 @@ parse_references (void *ctx,\r
+>> char *ref;\r
+>>\r
+>> if (refs == NULL || *refs == '\0')\r
+>> - return;\r
+>> + return NULL;\r
+>>\r
+>> while (*refs) {\r
+>> ref = _parse_message_id (ctx, refs, &refs);\r
+>> @@ -519,6 +521,8 @@ parse_references (void *ctx,\r
+>> if (ref && strcmp (ref, message_id))\r
+>> g_hash_table_insert (hash, ref, NULL);\r
+>> }\r
+>> +\r
+>> + return ref;\r
+>\r
+> As the comment for the function says, we explicitly avoid including\r
+> self-references. I think I'd err on the safe side and return NULL if the\r
+> last ref equals message-id.\r
+>\r
+>> }\r
+>>\r
+>> notmuch_status_t\r
+>> @@ -1365,7 +1369,7 @@ _notmuch_database_generate_doc_id (notmuch_database_t *notmuch)\r
+>> notmuch->last_doc_id++;\r
+>>\r
+>> if (notmuch->last_doc_id == 0)\r
+>> - INTERNAL_ERROR ("Xapian document IDs are exhausted.\n");\r
+>> + INTERNAL_ERROR ("Xapian document IDs are exhausted.\n");\r
+>\r
+> I don't know how you got this non-change hunk here, but please remove\r
+> it. :)\r
+>\r
+>>\r
+>> return notmuch->last_doc_id;\r
+>> }\r
+>> @@ -1509,7 +1513,7 @@ _notmuch_database_link_message_to_parents (notmuch_database_t *notmuch,\r
+>> const char **thread_id)\r
+>> {\r
+>> GHashTable *parents = NULL;\r
+>> - const char *refs, *in_reply_to, *in_reply_to_message_id;\r
+>> + const char *refs, *in_reply_to, *in_reply_to_message_id, *last_ref_message_id;\r
+>> GList *l, *keys = NULL;\r
+>> notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;\r
+>>\r
+>> @@ -1517,21 +1521,31 @@ _notmuch_database_link_message_to_parents (notmuch_database_t *notmuch,\r
+>> _my_talloc_free_for_g_hash, NULL);\r
+>>\r
+>> refs = notmuch_message_file_get_header (message_file, "references");\r
+>> - parse_references (message, notmuch_message_get_message_id (message),\r
+>> - parents, refs);\r
+>> + last_ref_message_id = parse_references (message,\r
+>> + notmuch_message_get_message_id (message),\r
+>> + parents, refs);\r
+>>\r
+>> in_reply_to = notmuch_message_file_get_header (message_file, "in-reply-to");\r
+>> parse_references (message, notmuch_message_get_message_id (message),\r
+>> parents, in_reply_to);\r
+>>\r
+>> - /* Carefully avoid adding any self-referential in-reply-to term. */\r
+>> in_reply_to_message_id = _parse_message_id (message, in_reply_to, NULL);\r
+>\r
+> I wonder if you should reuse your parse_references() change here, so\r
+> you'd set in_reply_to_message_id to the last message-id in\r
+> In-Reply-To. This might tackle some of the problematic cases directly,\r
+> but should still be all right per RFC 2822. I didn't verify how the\r
+> parser handles an RFC 2822 violating free form header though.\r
+\r
+Strike that based on http://www.jwz.org/doc/threading.html:\r
+\r
+"If there are multiple things in In-Reply-To that look like Message-IDs,\r
+only use the first one of them: odds are that the later ones are\r
+actually email addresses, not IDs."\r
+\r
+>> + /* If the parent message ID from the Reply-To and References\r
+>> + * headers are different, use the References one. This is because\r
+>> + * the Reply-To header is more likely to be in an non-standard\r
+>> + * format. */\r
+>> + if (in_reply_to_message_id &&\r
+>> + last_ref_message_id &&\r
+>> + strcmp (last_ref_message_id, in_reply_to_message_id)) {\r
+>> + in_reply_to_message_id = last_ref_message_id;\r
+>> + }\r
+\r
+I talked to Austin (CC) about the patch on IRC, and his comment was,\r
+perceptive as always:\r
+\r
+ 23:38 amdragon Is the logic in that patch equivalent to always using\r
+ the last message ID in references unless there is no\r
+ references header? Seems like it is, but in a\r
+ convoluted way.\r
+\r
+And that's actually the case, isn't it? To make the code reflect that,\r
+you should use last_ref_message_id, and if that's NULL, fallback to\r
+in_reply_to_message_id.\r
+\r
+> I suggest adding an else if branch (or revamp the above if condition) to\r
+> tackle the missing In-Reply-To header:\r
+>\r
+> else if (!in_reply_to_message_id && last_ref_message_id) {\r
+> in_reply_to_message_id = last_ref_message_id;\r
+> }\r
+\r
+Strike that, it should be the other way round.\r
+\r
+>\r
+>> + /* Carefully avoid adding any self-referential in-reply-to term. */\r
+>> if (in_reply_to_message_id &&\r
+>> strcmp (in_reply_to_message_id,\r
+>> notmuch_message_get_message_id (message)))\r
+>\r
+> If you change parse_references() to be careful about never returning a\r
+> self-reference, and set in_reply_to_message_id from there, I think you\r
+> can drop the strcmp here. And move the comment to an appropriate place.\r
+\r
+Strike that.\r
+\r
+> Thanks for the patch, I think we should do this. But this is an area\r
+> where I think we need to be careful, so another reviewer wouldn't\r
+> harm. Some tests for this would be good too, obviously.\r
+\r
+I got this part right. :)\r
+\r
+\r
+Jani.\r
+\r
+>\r
+>\r
+> BR,\r
+> Jani.\r
+>\r
+>> {\r
+>> _notmuch_message_add_term (message, "replyto",\r
+>> - _parse_message_id (message, in_reply_to, NULL));\r
+>> + in_reply_to_message_id);\r
+>> }\r
+>>\r
+>> keys = g_hash_table_get_keys (parents);\r
+>> --\r
+>> 1.8.1.4\r
+>> _______________________________________________\r
+>> notmuch mailing list\r
+>> notmuch@notmuchmail.org\r
+>> http://notmuchmail.org/mailman/listinfo/notmuch\r
projects / notmuch-archives.git / commitdiff