--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id DEBFA431FBD\r
+ for <notmuch@notmuchmail.org>; Sun, 23 Jun 2013 09:59:55 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id jC52PBMVKejM for <notmuch@notmuchmail.org>;\r
+ Sun, 23 Jun 2013 09:59:46 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-5.mit.edu (dmz-mailsec-scanner-5.mit.edu\r
+ [18.7.68.34])\r
+ by olra.theworths.org (Postfix) with ESMTP id AF04A431FAE\r
+ for <notmuch@notmuchmail.org>; Sun, 23 Jun 2013 09:59:46 -0700 (PDT)\r
+X-AuditID: 12074422-b7ef78e000000935-85-51c72981074c\r
+Received: from mailhub-auth-2.mit.edu ( [18.7.62.36])\r
+ by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 8F.FA.02357.18927C15; Sun, 23 Jun 2013 12:59:45 -0400 (EDT)\r
+Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
+ by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id r5NGxhZg030538; \r
+ Sun, 23 Jun 2013 12:59:44 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r5NGxeMv030643\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+ Sun, 23 Jun 2013 12:59:42 -0400\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1Uqndb-0004Kv-Oo; Sun, 23 Jun 2013 12:59:40 -0400\r
+Date: Sun, 23 Jun 2013 12:59:39 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Justus Winter <4winter@informatik.uni-hamburg.de>\r
+Subject: Re: header continuation issue in notmuch frontend/alot/pythons email\r
+ module\r
+Message-ID: <20130623165938.GA2214@mit.edu>\r
+References: <20130623131145.2526.439@thinkbox.jade-hamburg.de>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <20130623131145.2526.439@thinkbox.jade-hamburg.de>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFupnleLIzCtJLcpLzFFi42IRYrdT0W3UPB5o0LJH3GJ26w8mi+s3ZzI7\r
+ MHlMPH+azePZqlvMAUxRXDYpqTmZZalF+nYJXBmH+26wF/xTqdi8YQ9bA+NdmS5GTg4JAROJ\r
+ /q7ZbBC2mMSFe+uBbC4OIYF9jBJfX3xjgnA2MkrMXLEJyjnNJNF54wQ7hLOEUWLT/JVg/SwC\r
+ qhJPj91nBLHZBDQktu1fDmaLCJhKbHjwgB3EZhYwkri/YzpzFyMHh7BAmMTsTnuQMK+AtsSZ\r
+ MzvASoQE7CTWftzLBhEXlDg58wkLRKuWxI1/L5lAWpkFpCWW/+MACXMK2Es8v9/KCmKLCqhI\r
+ TDm5jW0Co9AsJN2zkHTPQuhewMi8ilE2JbdKNzcxM6c4NVm3ODkxLy+1SNdULzezRC81pXQT\r
+ IzisXZR2MP48qHSIUYCDUYmHN1P1WKAQa2JZcWXuIUZJDiYlUd6zUscDhfiS8lMqMxKLM+KL\r
+ SnNSiw8xSnAwK4nwbrgGVM6bklhZlVqUD5OS5mBREucVu7UzUEggPbEkNTs1tSC1CCYrw8Gh\r
+ JMFrqQE0VLAoNT21Ii0zpwQhzcTBCTKcB2j4GnWgGt7igsTc4sx0iPwpRl2OyWe3vGcUYsnL\r
+ z0uVEuddrAZUJABSlFGaBzcHlo5eMYoDvSXMuwFkFA8wlcFNegW0hAloyZ7Vh0CWlCQipKQa\r
+ GJs3HSk6037wGUfJAp1DjdMTuO7ZqRlN5Thmv5HpYuz3JXwpOe0eblUTNyYumnVHc52RI+PJ\r
+ tQZxncq/2L/9WDb5Yq+V+fFlV7gfxLYsVlCLzNrhl1kUUriojv/x2xu3Lqny62Q0WC4Sv3C8\r
+ /IXYbdP3JVnpQkvvf4++dHSLJesak6OMKy4zXVFiKc5INNRiLipOBADxp15XIgMAAA==\r
+Cc: notmuch mailing list <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 23 Jun 2013 16:59:56 -0000\r
+\r
+Quoth Justus Winter on Jun 23 at 3:11 pm:\r
+> Hi,\r
+> \r
+> I recently had a problem replying to a mail written by Thomas Schwinge\r
+> using an oldish notmuch. Not sure if it has been fixed in more recent\r
+> versions, but I think notmuch could improve uppon its header\r
+> generation (see below). Problematic part of the mail:\r
+> \r
+> ~~~ snip ~~~\r
+> [...]\r
+> To: someone@example.org, "line\r
+> break" <linebreak@example.org>, someoneelse@example.org\r
+> User-Agent: Notmuch/0.9-101-g81dad07 (http://notmuchmail.org) Emacs/23.4.1 (i486-pc-linux-gnu)\r
+> [...]\r
+> ~~~ snap ~~~\r
+> \r
+> http://tools.ietf.org/html/rfc2822#section-2.2.3 says:\r
+> \r
+> Note: Though structured field bodies are defined in such a way that\r
+> folding can take place between many of the lexical tokens (and even\r
+> within some of the lexical tokens), folding SHOULD be limited to\r
+> placing the CRLF at higher-level syntactic breaks. For instance, if\r
+> a field body is defined as comma-separated values, it is recommended\r
+> that folding occur after the comma separating the structured items in\r
+> preference to other places where the field could be folded, even if\r
+> it is allowed elsewhere.\r
+> \r
+> So notmuch "rfc-SHOULD" place the newlines after the comma.\r
+> \r
+> The rfc goes on:\r
+> \r
+> The process of moving from this folded multiple-line representation\r
+> of a header field to its single line representation is called\r
+> "unfolding". Unfolding is accomplished by simply removing any CRLF\r
+> that is immediately followed by WSP. Each header field should be\r
+> treated in its unfolded form for further syntactic and semantic\r
+> evaluation.\r
+> \r
+> My interpretation is that unfolding simply removes any linebreaks\r
+> first, so the value does not contain any newlines. But pythons email\r
+> module discriminates quoted and unquoted parts of the value:\r
+> \r
+> ~~~ snip ~~~\r
+> from __future__ import print_function\r
+> import email\r
+> from email.utils import getaddresses\r
+> \r
+> m = email.message_from_string('''To: "line\r
+> break" <linebreak@example.org>, line\r
+> break <linebreak@example.org>''')\r
+> print("m['To'] = ", m['To'])\r
+> print("getaddresses(m.get_all('To')) = ", getaddresses(m.get_all('To')))\r
+> ~~~ snap ~~~\r
+> \r
+> % python3 test.py\r
+> m['To'] = "line\r
+> break" <linebreak@example.org>, line\r
+> break <linebreak@example.org>\r
+> getaddresses(m.get_all('To')) = [('line\n break', 'linebreak@example.org'), ('line break', 'linebreak@example.org')]\r
+> \r
+> I believe that is what's preventing me from replying to the message\r
+> using alot without sanitizing the To header first. Not really sure who\r
+> is wrong or right here... any thoughts?\r
+\r
+There are at least two bugs here. Regardless of what we RFC-should\r
+do, that folding *is* permitted by RFC2822, since quoted\r
+strings can contain folding whitespace:\r
+\r
+ http://tools.ietf.org/html/rfc2822#section-3.2.5\r
+\r
+For completeness, the full derivation for this "To" header is:\r
+\r
+to = "To:" address-list CRLF\r
+address-list = (address *("," address)) / obs-addr-list\r
+address = mailbox / group\r
+mailbox = name-addr / addr-spec\r
+name-addr = [display-name] angle-addr\r
+display-name = phrase\r
+phrase = 1*word / obs-phrase\r
+word = atom / quoted-string\r
+quoted-string = [CFWS]\r
+ DQUOTE *([FWS] qcontent) [FWS] DQUOTE\r
+ [CFWS]\r
+\r
+Do you happen to know how the strangely folded "to" header was\r
+produced for this message? In notmuch-emacs, a user can put whatever\r
+they want in a message-mode buffer's headers and mm will dutifully\r
+pass it on to their MTA. We could validate it, but that's a slippery\r
+slope and I would hope that the MTA itself is validating it (and\r
+probably more thoroughly than we could).\r
+\r
+That said, the first bug here is in Python. As I mentioned above,\r
+foldable whitespace is allowed in quoted strings. In fact, though the\r
+standard is rather long-winded about whitespace, if you dig into the\r
+grammar, you'll find that *all whitespace can be folded* (except in\r
+the obsolete grammar, which allowed whitespace between the header name\r
+and the colon, which obviously can't be folded). I'm not sure what\r
+Python is doing, but I bet it's going to a lot of effort to\r
+mis-implement something very simple.\r
+\r
+There also appears to be a bug in the notmuch CLI's reply command\r
+where it omits addresses that were folded in the original message. I\r
+don't know if alot uses the CLI's reply command, so this may or may\r
+not be related to your specific issue. I haven't dug into this yet,\r
+other than to confirm that it's the CLI's fault and not\r
+notmuch-emacs's.\r
+\r
+> Justus\r
projects / notmuch-archives.git / commitdiff