Date: Tue, 11 Feb 2014 12:11:35 -0800
From: "W. Trevor King" <wking@tremily.us>
To: Tomi Ollila <tomi.ollila@iki.fi>
Subject: Re: [PATCH v2 14/20] nmbug-status: Encode output using the user's
	locale
Message-ID: <20140211201135.GJ14197@odin.tremily.us>
References: <cover.1392056624.git.wking@tremily.us>
	<deff072f78f4a7c5b0774e67a8f0517cc704725d.1392056624.git.wking@tremily.us>
	<87eh396e6e.fsf@zancas.localnet>
	<m2sirpu46i.fsf@guru.guru-group.fi>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="MR4jz7xdnY3JMfbc"
Content-Disposition: inline
In-Reply-To: <m2sirpu46i.fsf@guru.guru-group.fi>
OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8;
	url=http://tremily.us/pubkey.txt
User-Agent: Mutt/1.5.22 (2013-10-16)
Cc: notmuch@notmuchmail.org
Precedence: list


--MR4jz7xdnY3JMfbc
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 11, 2014 at 04:14:45PM +0200, Tomi Ollila wrote:
> On Tue, Feb 11 2014, David Bremner wrote:
> > W. Trevor King writes:
> >> Instead of always writing UTF-8, allow the user to configure the
> >> output encoding using their locale.  This is useful for
> >> previewing output in the terminal, for poor souls that don't use
> >> UTF-8 locales ;).
> >
> > =E2=80=A6
> > remote: UnicodeEncodeError: 'ascii' codec can't encode character
> >   u'\u017b' in position 219: ordinal not in range(128)
> >
> > possibly because of
> >
> > LANG=3DC
> > =E2=80=A6
> >
> > I think it's fine to _allow_ the user to configure the output
> > encoding. I'm less sure about _requiring_ it.

If a user has set LANG=3DC, I expect that's what we should use for
output (in which case dying with an encoding error is the right thing
to do).  If you want UTF-8 output, using a UTF-8 locale seems like a
reasonable requirement.  For the HTML case, we could fall back on
numerical character references (e.g. &#x017b;) if the requested locale
didn't support the required character directly, but I don't see an
easy solution for the text-mode output.

> That reminded me that yesterday (after review, of course) I thought
> that we probably want configuration file to be parsed as utf-8
> instead of any encoding user may have in their system...

The POSIX spec for LANG doesn't restrict the scoping to the terminal
intput / output [1], so I feel like we should also be using LANG to
read the config file as well.  I expect folks with UTF-8 LANGs will
want UTF-8 file contents.  In both cases (terminal output and
config-file input), it is easy for users to pick their preferred
encoding:

  $ LANG=3Den_US.UTF-8 nmbug-status =E2=80=A6

I think we should trust what they've chosen, rather than guessing that
they actually want UTF-8 ;).

Cheers,
Trevor

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.htm=
l#tag_08_02

--=20
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

--MR4jz7xdnY3JMfbc
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJS+oP0AAoJEKKfehoaNkbt6ZAP/0H+pnE5bI6Fr1nukjMFq93W
OzbHgIHma+iXHjW4++gid1C1upuDVqZ7n4OkMzRf4sLoh9YRnNIvTaoYyiNUlceN
tiNOzTSNI5r/AJpQKP1pOQIySRMNPbBAvxbEHoL7FhQyOOvPlDR2wuLJw0Scg577
6bksYfSiOoxYYPfNb/Nt0PUYBAL3BWgNyH/47DPdqANgZK7OhY2KuzEwBiBChE4l
cW81OlnqiczYPmSfxLjxLHDyJIgDZh7AN++UgXndLtoUy1xF31pIoJFF5KMdF5EU
SaI3zWqrqA4CzZEWUIzTo6VwqCJj3AMzaGN02R/jUrcunfgmSsjoqiPyLGaoU9xr
eFhFUuPTWr9jPqevaPmutUaj2mp2KfSPzv4ImJCsQZIdO/L91ZCVlzt7nVRxOpp4
l50gfHFP0BGiGwFqP+obZIEvTisJII1bIIIvQvVXKdOHIi0aOcrNzrlvS6VCy3E+
P2zxj/PMPApvXisuMybDhpjYrogxwyYREdCpgd3601VXHXVIFicVEgiy5g0AfhIv
1U0l4xUpgcahbf7gNFTV+NigxHBXvJXBGSnAelu5mTACY6TeK6Sw0uDfsb0QzwXS
8XlcR4tol6Sv8+tsgbTaSXiX3LtWDT6BSsq7+cTRVwD/9oebNsgPwK/GrNuCkAEs
fGIcff1XpR4iPmygM/xn
=A9fT
-----END PGP SIGNATURE-----

--MR4jz7xdnY3JMfbc--