Re: [PATCH] mime_node_open: skip envelope from lines at the start of messages

diff --git a/d7/a482764c26d7432e9002590d40ad965c42c19a b/d7/a482764c26d7432e9002590d40ad965c42c19a
new file mode 100644 (file)
index 0000000..59b4e13
--- /dev/null
+++ b/d7/a482764c26d7432e9002590d40ad965c42c19a
@@ -0,0 +1,94 @@
+Return-Path: <dkg@fifthhorseman.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 764C9431FB6\r
+       for <notmuch@notmuchmail.org>; Fri,  9 Mar 2012 08:20:06 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id cyu-4A2-tyRv for <notmuch@notmuchmail.org>;\r
+       Fri,  9 Mar 2012 08:20:05 -0800 (PST)\r
+Received: from che.mayfirst.org (che.mayfirst.org [209.234.253.108])\r
+       by olra.theworths.org (Postfix) with ESMTP id CBEE2431FAE\r
+       for <notmuch@notmuchmail.org>; Fri,  9 Mar 2012 08:20:05 -0800 (PST)\r
+Received: from [192.168.23.207] (dsl254-070-154.nyc1.dsl.speakeasy.net\r
+       [216.254.70.154])\r
+       by che.mayfirst.org (Postfix) with ESMTPSA id 102AFF970;\r
+       Fri,  9 Mar 2012 11:20:01 -0500 (EST)\r
+Message-ID: <4F5A2DB1.7040301@fifthhorseman.net>\r
+Date: Fri, 09 Mar 2012 11:20:01 -0500\r
+From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>\r
+User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20120125 Icedove/9.0.1\r
+MIME-Version: 1.0\r
+To: David Bremner <david@tethera.net>, notmuch <notmuch@notmuchmail.org>\r
+Subject: Re: [PATCH] mime_node_open: skip envelope from lines at the start\r
+       of messages\r
+References: <m2aa3qjbfo.fsf@guru.guru-group.fi>\r
+       <1331301387-23630-1-git-send-email-david@tethera.net>\r
+In-Reply-To: <1331301387-23630-1-git-send-email-david@tethera.net>\r
+Content-Type: text/plain; charset=UTF-8; format=flowed\r
+Content-Transfer-Encoding: 7bit\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 09 Mar 2012 16:20:06 -0000\r
+\r
+On 03/09/2012 08:56 AM, David Bremner wrote:\r
+> Some MDAs such as procmail (in MH mode), and exim (doing local\r
+> delivery in some configurations of the appendfile transport) add a\r
+> line to the front of a message with "From " followed by envelope\r
+> sender.  Since this is not a proper RFC822 header field, gmime (at\r
+> least since version 2.6) refuses to parse it, unless in mbox mode.\r
+>\r
+> This change reads the line of the file, and if they start with\r
+> "From ", pass the stream to gmime starting from the second line.\r
+>\r
+> This makes mime_node_open more consistent with (but still stricter\r
+> than) the permissive behaviour of notmuch_file_get_header\r
+> (message-file.c), which allows a certain number of "broken_headers".\r
+>\r
+> We avoid putting gmime into mbox mode in case of side effects; this\r
+> leaves the situation of mboxes accidentally indexed by notmuch the\r
+> same as before, namely "undefined behaviour".  Ideally they should at\r
+> least be warned by notmuch-new.  Although strict rfc822 adherence\r
+> would be one way to detect mboxes, it doesn't seem to fit with the\r
+> spirit or code of message-file.c.\r
+\r
+The above justification (and the version of the associated patch without \r
+the memory leak and using strncmp instead of strcmp) seems good to me.\r
+\r
+While I'd prefer to have nothing but spic-and-span, perfectly clean \r
+RFC2822 messages, we have (perhaps accidentally) traditionally supported \r
+message files with leading "From " lines, so they will be \r
+already-indexed by previous versions of notmuch.\r
+\r
+This patch defines the non-MIME variance we're willing to accept quite \r
+narrowly (just a single leading line that starts with "From ", no \r
+escaping of the rest of the text), avoids breaking compatibility with \r
+existing indexes, and satisfies indexing some plausible MTA delivery \r
+configurations.\r
+\r
+The only way it would be better is if it were to auto-detect that a file \r
+is actually a multi-message mbox, and alert the user to the fact that \r
+all but the first message in the mbox is unindexed.  But we don't \r
+currently do that anyway, so it's not a regression (and that additional \r
+cleanup should probably be a separate patch anyway).\r
+\r
+so: LGTM.\r
+\r
+       --dkg\r