From: Ethan Glasser-Camp <ethan.glasser.camp@gmail.com>
To: Tomi Ollila <tomi.ollila@iki.fi>,
	Michael Stapelberg <michael+nm@stapelberg.de>, notmuch@notmuchmail.org
Subject: Re: [BUG] Saving attachments containing UTF-8 chars
In-Reply-To: <m2pq416xzl.fsf@guru.guru-group.fi>
References: <x6sj8xiexi.fsf@midna.zekjur.net>
	<m2pq416xzl.fsf@guru.guru-group.fi>
User-Agent: Notmuch/0.14+45~g6ea9330 (http://notmuchmail.org) Emacs/24.1.1
	(x86_64-pc-linux-gnu)
Date: Sun, 18 Nov 2012 00:29:31 -0500
Message-ID: <87zk2f1nt0.fsf@betacantrips.com>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list

Tomi Ollila <tomi.ollila@iki.fi> writes:

> I can verify this bug: I copied 'rawmail' to my mail store and attempted
> to 'w' the attacment and got the same result (after notmuch new).
>
> The saving code first does
> notmuch show --format=raw id:"508953E6.70006@gmail.com"
> which decodes OK on command line, and to the buffer when 
> kill-buffer is outcommented in (with-current-notmuch-show-message ...) 
> macro.

I was able to see this behavior, and Tomi did a good job tracking down
where it was :)

I even see the bytes as presented in the file. When moving point to the
problematic character, and doing M-x describe-char, it says:

          buffer code: #xE2 #x80 #x99
            file code: #xE2 #x80 #x99 (encoded by coding system utf-8)

buffer-file-coding-system is, of course, utf-8.

Writing this buffer using C-x C-w encodes it correctly too. So I think
this is an emacs MIME problem. We call mm-save-part, which calls
mm-save-part-to-file, which calls mm-with-unibyte-buffer. Hmm..

Indeed, it seems that inserting this character into a file that's been
marked "unibyte" using (set-buffer-multibyte nil) turns it into the ^Y
character (ASCII code 0x19 -- the character that comes out in the patch
file). There's probably a technical reason that this should be true, but
I can't think of why that would be.

> I attempted a set of trial-&-error tricks to get the attachment
> saved "correctly", and at least this seems to do the trick:
>
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index f273eb4..a6a85c0 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -203,9 +203,11 @@ For example, if you wanted to remove an \"unread\" tag and add a
>       (let ((id (notmuch-show-get-message-id)))
>         (let ((buf (generate-new-buffer (concat "*notmuch-msg-" id "*"))))
>           (with-current-buffer buf
> -	    (call-process notmuch-command nil t nil "show" "--format=raw" id)
> -           ,@body)
> -	 (kill-buffer buf)))))
> +	   (let ((coding-system-for-read 'no-conversion)
> +		 (coding-system-for-write 'no-conversion))
> +	     (call-process notmuch-command nil t nil "show" "--format=raw" id)
> +	     ,@body))))))
> +%%	 (kill-buffer buf)))))
[snip]
> (kill-buffer is outcommented above for testing purposes)
>
> To test this this needs to me evaluated and then the functions
> using this macro (notmuch-show-save-attachments  in this case)
>
> Smart suggestions for proper fix ?

Well, we could limit it just to saving attachments (putting the let
around the with-current-notmuch-show-message). That feels like it could
be right, because intuitively saving an attachment should be done
without any conversions. Or even the above doesn't seem so bad. My vague
feeling is that messages should always be ASCII, or at least mm-* will
interpret it that way, so decoding them into any other character set
might cause problems. Anyone understand character sets?

Ethan