Date: Tue, 4 Feb 2014 11:14:53 -0800
From: "W. Trevor King" <wking@tremily.us>
To: Tomi Ollila <tomi.ollila@iki.fi>
Subject: Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general
	refactoring
Message-ID: <20140204191453.GV14197@odin.tremily.us>
References: <cover.1391424512.git.wking@tremily.us>
	<m2ha8frjgg.fsf@guru.guru-group.fi>
	<20140204005331.GQ14197@odin.tremily.us>
	<m2fvnzp3uh.fsf@guru.guru-group.fi>
	<20140204161142.GS14197@odin.tremily.us>
	<m2a9e6zppp.fsf@guru.guru-group.fi>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="zD4kWSsXNMryRTZe"
Content-Disposition: inline
In-Reply-To: <m2a9e6zppp.fsf@guru.guru-group.fi>
OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8;
	url=http://tremily.us/pubkey.txt
User-Agent: Mutt/1.5.22 (2013-10-16)
Cc: notmuch@notmuchmail.org
Precedence: list


--zD4kWSsXNMryRTZe
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 04, 2014 at 08:40:18PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, W. Trevor King wrote:
> >
> >   >>> from __future__ import unicode_literals
> >   >>> import codecs
> >   >>> import locale
> >   >>> import sys
> >   >>> print(locale.getpreferredencoding())  # same as yours
> >   UTF-8
> >   >>> print(sys.getdefaultencoding())  # same as yours
> >   ascii
> >   >>> _ENCODING =3D locale.getpreferredencoding() or sys.getdefaultenco=
ding()
> >   >>> print(_ENCODING)  # double-check default encodings
> >   UTF-8
> >   >>> byte_stream =3D sys.stdout  # copied from Page.write
> >   >>> stream =3D codecs.getwriter(encoding=3D_ENCODING)(stream=3Dbyte_s=
tream)
> >   >>> data =3D {'from': '\u017b'}  # fake the troublesome data
> >   >>> print(type(data['from']))  # double-check unicode_literals
> >   <type 'unicode'>
> >   >>> string =3D '  <td>{from}</td>\n'.format(**data)
> >   >>> stream.write(string)
> >     <td>=C5=BB</td>
> >
> > It looks like you'll have the same _ENCODING as I do (UTF-8).  That
> > means your stream should be wrapped in a UTF-8 StreamWriter, so I
> > don't understand why it's converting to ASCII.  Can you run through
> > the above on your troublesome machine and confirm that stream.write()
> > is still raising the exception?  If it doesn't work, can you just
> > paste that whole run in your next email?
>=20
> I don't know what to paste, so i paste this:
>=20
> $ python
> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)=20
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.

It looks like you left out:

  from __future__ import unicode_literals

Can you try again with that line as the first command?

> >>> data =3D {'from': '\u017b'}
> >>> print(type(data['from']))=20
> <type 'str'>

which is why your data is a 'str' and not a 'unicode' instance.

> >>> string =3D '  <td>{from}</td>\n'.format(**data)
> >>> print string
>   <td>\u017b</td>
>=20
> and then:
>=20
> >>> data =3D {'from': u'\u017b'}

This works around the lack of unicode_literals with an explicit u''.

> >>> print(type(data['from']))=20
> <type 'unicode'>
> >>> string =3D '  <td>{from}</td>\n'.format(**data)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in

However, without unicode_literals or an explicit u'', you're format
string '=E2=80=A6{from}' is a str (it should be a 'unicode' instance with
unicode_literals).

> >>> import os
> >>> print os.environ['LANG']
> en_US.UTF-8

That's good anyway ;).  Thanks for digging into this :).

Cheers,
Trevor

--=20
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

--zD4kWSsXNMryRTZe
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJS8TwsAAoJEKKfehoaNkbtRPAP/i/PtIT6gMW14hkLuPy1BksK
m7TuPzNokEAE0+RVQF2OIFrK3VboowMgQZICL70ZUIYZmp3vSyJv4m9SWH2N2gw9
0USZhga1OYUmS3eUucFjfuCK5dVXYOxUtRvXoUZh3AgHjV0V6bIu0QpTCJghEKSG
Z5k4Bway4dV/CjNFQsHFzqpsw3W93+eOWQdi6ddV0v2yzSc7h8eVlB/i2DHcvSXX
VjHwPnKVt8UnEWUf9MyYGrSDzUfJrmr60ux4ntTykHNvlDV7gX2I1yHtx/K/iwks
n31fARQmM3F7r3sAkM4IWHT/+bjRfD+k5S2EnQDdNadR1T0etO/+ybrFZwTB8nmV
gwdHdOylsJcqiFl9y2tbplVTdxnAKy8y1di2zxYCEGNL4BhWv2ak8wFz9dMkAs9W
hUI6Q0GcCR0E9BWksts6FJYP+ZgoFv4tKrozW++5VVy0gExQPv8bF4wOngCnibqk
Ir7Ec6K88N4Az320xdTzQZR+nr9i272f5Aa4XOOFfEJpZQrH1eIe5Eiu0jymraen
SybaBMGDlAy4D4c7FXJ6XlPWolUH1nw8fVq3eoDq5qN5/b6F05cb8ouP1vG2q8fn
dg2QllsAD6GB001nzxCMo5JlwBdTw/l78f2aujeK7L6toWSWWBfgKNfddZWq6CKu
o52ndagLxMGPpKoY15og
=gPZZ
-----END PGP SIGNATURE-----

--zD4kWSsXNMryRTZe--