--- /dev/null
+Return-Path: <tomi.ollila@iki.fi>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 4EB256DE1862\r
+ for <notmuch@notmuchmail.org>; Mon, 14 Mar 2016 00:23:28 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0.634\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0.634 tagged_above=-999 required=5 tests=[AWL=-0.018,\r
+ SPF_NEUTRAL=0.652] autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id ljyua7JZvy8y for <notmuch@notmuchmail.org>;\r
+ Mon, 14 Mar 2016 00:23:25 -0700 (PDT)\r
+Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])\r
+ by arlo.cworth.org (Postfix) with ESMTP id E3CA66DE185F\r
+ for <notmuch@notmuchmail.org>; Mon, 14 Mar 2016 00:23:24 -0700 (PDT)\r
+Received: from guru.guru-group.fi (localhost [IPv6:::1])\r
+ by guru.guru-group.fi (Postfix) with ESMTP id 97075100063;\r
+ Mon, 14 Mar 2016 09:23:21 +0200 (EET)\r
+From: Tomi Ollila <tomi.ollila@iki.fi>\r
+To: Jani Nikula <jani@nikula.org>, Edward Betts <edward@4angle.com>,\r
+ notmuch@notmuchmail.org\r
+Subject: Re: [PATCH] Don't bother checking for mbox files\r
+In-Reply-To: <87mvq2pmqe.fsf@nikula.org>\r
+References: <86io0v9oum.fsf@hiro.keithp.com>\r
+ <20160313105742.GA9173@4angle.com> <87mvq2pmqe.fsf@nikula.org>\r
+User-Agent: Notmuch/0.21+81~g4743a61 (http://notmuchmail.org) Emacs/24.3.1\r
+ (x86_64-unknown-linux-gnu)\r
+X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL\r
+ $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F\r
+ !)g;OY^,BjTbr)Np:%c_o'jj,Z\r
+Date: Mon, 14 Mar 2016 09:23:21 +0200\r
+Message-ID: <m2io0pzfo6.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 14 Mar 2016 07:23:28 -0000\r
+\r
+On Sun, Mar 13 2016, Jani Nikula <jani@nikula.org> wrote:\r
+\r
+> [ text/plain ]\r
+> On Sun, 13 Mar 2016, Edward Betts <edward@4angle.com> wrote:\r
+>> Keith Packard <keithp@keithp.com> wrote:\r
+>>> Postfix adds mbox-style From lines when used in combination with\r
+>>> maildrop or .forward files. If they have another line starting with\r
+>>> 'From ' in them, notmuch complains about them not being mail files.\r
+>>> \r
+>>> If we assume the user hasn't screwed up and misconfigured their mail\r
+>>> system, then we can safely ignore whether the file started with an\r
+>>> mbox header and just parse it as a single-message file.\r
+>>\r
+>> I think it is fine to go ahead with this change. At the same time the\r
+>> behaviour of Postfix should be corrected so it doesn't add mbox-style From\r
+>> lines to mails in maildir format.\r
+>\r
+> I disagree with making the change (as-is, at least).\r
+>\r
+> In general, Notmuch does not support mboxes. We expect maildir style one\r
+> message per file mail storage. We support single-message mboxes as a\r
+> special case, in part because, as you note, there's plenty of other\r
+> software that adds the mbox "From " line even though delivering to\r
+> maildir.\r
+>\r
+> I think it's misleading and confusing to the users to accept and index\r
+> the first message of mboxes, and silently ignore the rest (or worse,\r
+> index all of the mbox and associate the text with the first message). I\r
+> think we should reject multi-message mboxes, because we have no code to\r
+> handle them. This patch throws away that check.\r
+>\r
+> Now, IIUC, the problem here is not that the files actually are\r
+> multi-message mboxes. We could use a sample message (even a crafted one)\r
+> that exhibits the problem, so we could add a test case, and fix Notmuch\r
+> to deal with it gracefully (if we decide catering to potentially broken\r
+> other software is the way to go), while retaining the code to reject\r
+> multi-message mboxes. With the test case, we'd also avoid accidentally\r
+> breaking this in the future.\r
+\r
+I agree with Jani; user may accidentally index one mbox with multiple\r
+messages as single message if this were merged...\r
+\r
+We currently have very simple check; just line starting with 'From ' to\r
+separate messages (and first line starts with 'From '). After a quick check\r
+of these 'mbox*' "specs" this may just be within the "standard".\r
+\r
+In mboxviewfs I checked whether there is at least one empty line before\r
+'^From' (might not be required by the standard, but whatever ;/) and that\r
+there is at least 'Date:' header following (needed for file "time")... but\r
+even this "heuristics" may not be enough if we wanted to go deep into\r
+this (i.e. there are emails which quote beginning of an mbox file (ok, no\r
+heuristics can match this unless there is human-level AI working on it ;)\r
+\r
+OTOH, presumably\r
+\r
+https://github.com/GNOME/gmime/blob/master/tests/data/mbox/input/substring.mbox\r
+\r
+contains 3 messages (or what??!!11)\r
+\r
+...\r
+\r
+Perhaps the simplest is to give users possibility to use 'footgun' option\r
+in notmuch new (notmuch insert probably doesn't need it ???) which can be\r
+used to skip the 'mbox' check (I was going to suggest configuration option,\r
+but as we don't support that in bindings, ...). But of course some of the\r
+simplicity is gone when one forgets to give the --footgun option -- next\r
+notmuch new with the footgun probably will not pick the mail file again\r
+(or we have to hold on updating the directory mtime indefinitely -- or\r
+do other changes (i.e. more complicated which no-one reviews(*) anyway >;/))\r
+\r
+\r
+> BR,\r
+> Jani.\r
+\r
+Tomi\r
+\r
+(*) Although when someone sends less than usual trivial patches which\r
+provides significant progression to the functionality those are reviewed\r
+promptly with a relatively good number of reviewers...\r
+\r
+One 'other change' could be e.g. keep a list of files that has been failing\r
+due to this and retry those if this footgun option is given.\r
projects / notmuch-archives.git / commitdiff