--- /dev/null
+Return-Path: <sojka@os.inf.tu-dresden.de>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id E6C48431FB6\r
+ for <notmuch@notmuchmail.org>; Tue, 22 May 2012 05:53:35 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 6EZeAtj3uhqd for <notmuch@notmuchmail.org>;\r
+ Tue, 22 May 2012 05:53:34 -0700 (PDT)\r
+Received: from max.feld.cvut.cz (max.feld.cvut.cz [147.32.192.36])\r
+ by olra.theworths.org (Postfix) with ESMTP id 9C76B431FAE\r
+ for <notmuch@notmuchmail.org>; Tue, 22 May 2012 05:53:34 -0700 (PDT)\r
+Received: from localhost (unknown [192.168.200.4])\r
+ by max.feld.cvut.cz (Postfix) with ESMTP id 2D88919F3345;\r
+ Tue, 22 May 2012 14:53:33 +0200 (CEST)\r
+X-Virus-Scanned: IMAP AMAVIS\r
+Received: from max.feld.cvut.cz ([192.168.200.1])\r
+ by localhost (styx.feld.cvut.cz [192.168.200.4]) (amavisd-new,\r
+ port 10044)\r
+ with ESMTP id AYx5Np-t01_W; Tue, 22 May 2012 14:53:28 +0200 (CEST)\r
+Received: from imap.feld.cvut.cz (imap.feld.cvut.cz [147.32.192.34])\r
+ by max.feld.cvut.cz (Postfix) with ESMTP id 4839919F331A;\r
+ Tue, 22 May 2012 14:53:28 +0200 (CEST)\r
+Received: from steelpick.2x.cz (note-sojka.felk.cvut.cz [147.32.86.30])\r
+ (Authenticated sender: sojkam1)\r
+ by imap.feld.cvut.cz (Postfix) with ESMTPSA id 2164E660968;\r
+ Tue, 22 May 2012 14:53:27 +0200 (CEST)\r
+Received: from wsh by steelpick.2x.cz with local (Exim 4.77)\r
+ (envelope-from <sojka@os.inf.tu-dresden.de>)\r
+ id 1SWoac-0000JS-V4; Tue, 22 May 2012 14:53:26 +0200\r
+From: Michal Sojka <sojkam1@fel.cvut.cz>\r
+To: Adam Wolfe Gordon <awg+notmuch@xvx.ca>, Tomi Ollila <tomi.ollila@iki.fi>\r
+Subject: Re: emacs complains about encoding?\r
+In-Reply-To:\r
+ <CAMoJFUungAFPWy0d1Lh+rqmpK--P7MMEwNaewWHR=rbYo+BKsA@mail.gmail.com>\r
+References: <20120515194455.B7AD5100646@guru.guru-group.fi>\r
+ <878vgsbprq.fsf@nikula.org> <m23970bhre.fsf@guru.guru-group.fi>\r
+ <CAMoJFUungAFPWy0d1Lh+rqmpK--P7MMEwNaewWHR=rbYo+BKsA@mail.gmail.com>\r
+User-Agent: Notmuch/0.12+185~g9826d2c (http://notmuchmail.org) Emacs/23.4.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Tue, 22 May 2012 14:53:26 +0200\r
+Message-ID: <871umc1int.fsf@steelpick.2x.cz>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 22 May 2012 12:53:36 -0000\r
+\r
+Hello Adam,\r
+\r
+Adam Wolfe Gordon <awg+notmuch@xvx.ca> writes:\r
+> It turns out it's actually not the emacs side, but an interaction\r
+> between our JSON reply format and emacs.\r
+>\r
+> The JSON reply (and show) code includes part content for all text/*\r
+> parts except text/html. Because all JSON is required to be UTF-8, it\r
+> handles the encoding itself, puts UTF-8 text in, and omits a\r
+> content-charset field from the output. Emacs passes on the\r
+> content-charset field to mm-display-part-inline if it's available, but\r
+> for text/plain parts it's not, leaving mm-display-part-inline to its\r
+> own devices for figuring out what the charset is. It seems\r
+> mm-display-part-inline correctly figures out that it's UTF-8, and puts\r
+> in the series of ugly \nnn characters because that's what emacs does\r
+> with UTF-8 sometimes.\r
+>\r
+> In the original reply stuff (pre-JSON reply format) emacs used the\r
+> output of notmuch reply verbatim, so all the charset stuff was handled\r
+> in notmuch. Before f6c170fabca8f39e74705e3813504137811bf162, emacs was\r
+> using the JSON reply format, but was inserting the text itself instead\r
+> of using mm-display-part-inline, so emacs still wasn't trying to do\r
+> any charset manipulation. Using mm-display-part-inline is desirable\r
+> because it lets us handle non-text/plain (e.g. text/html) parts\r
+> correctly in reply, and makes the display more consistent (since we\r
+> use it for show). But, it leads to this problem.\r
+>\r
+> So, there are a couple of solutions I can see:\r
+>\r
+> 1) Have the JSON formats include the original content-charset even\r
+> though they're actually outputting UTF-8. Of the solutions I tried,\r
+> this is the best, even though it doesn't sound like a good thing to\r
+> do.\r
+>\r
+> 2) Have the JSON formats include content only if it's actually UTF-8.\r
+> This means that for non-UTF-8 parts (including ASCII parts), the emacs\r
+> interface has to do more work to display the part content, since it\r
+> must fetch it from outside first. When I tried this, it worked but\r
+> caused the \nnn to show up when viewing messages in emacs. I suspect\r
+> this is because it sets a charset for the whole buffer, and can't\r
+> accommodate messages with different charsets in the same buffer\r
+> properly. Reply works correctly, though.\r
+>\r
+> 3) Have the JSON formats include the charset for all parts, but make\r
+> it UTF-8 for all parts they include content for (since we're actually\r
+> outputting UTF-8). This doesn't seem to fix the problem, even though\r
+> it seems like it should.\r
+>\r
+> If no one has a better idea or a strong reason not to, I'll send a\r
+> patch for solution (1).\r
+\r
+Thank you very much for your analysis. It encouraged me to dig into the\r
+problem and I've found another solution, which might be better than\r
+those you suggested.\r
+\r
+I traced what Emacs does with the text inside\r
+notmuch-mm-display-part-inline and the wrong charset conversion happens\r
+deeply in elisp code in mm-with-part called by mm-get-part, which is in\r
+turn called by mm-inline-text. There is a way to make mm-inline-text not\r
+to call mm-get-part, which is to set the charset to 'gnus-decoded. This\r
+sounds like something that applies to our situation, where the part is\r
+already decoded.\r
+\r
+The following patch (apply it with git am -c) solves the problem for me.\r
+However, I'm not sure it is a universal solution. It sets the charset\r
+only if it is not defined in notmuch json output and I'm not sure that\r
+this is correct. text/html parts seem to have charset defined, but as\r
+you wrote that json is always utf-8, so it might be that we need\r
+'gnus-decoded always, independently of the json output. What do you\r
+think?\r
+\r
+-Michal\r
+\r
+----8<-------\r
+diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el\r
+index 7fa441a..8070f05 100644\r
+--- a/emacs/notmuch-lib.el\r
++++ b/emacs/notmuch-lib.el\r
+@@ -244,7 +244,7 @@ the given type."\r
+ current buffer, if possible."\r
+ (let ((display-buffer (current-buffer)))\r
+ (with-temp-buffer\r
+- (let* ((charset (plist-get part :content-charset))\r
++ (let* ((charset (or (plist-get part :content-charset) 'gnus-decoded))\r
+ (handle (mm-make-handle (current-buffer) `(,content-type (charset . ,charset)))))\r
+ ;; If the user wants the part inlined, insert the content and\r
+ ;; test whether we are able to inline it (which includes both\r
projects / notmuch-archives.git / commitdiff