Date: Tue, 4 Feb 2014 08:11:42 -0800
From: "W. Trevor King" <wking@tremily.us>
To: Tomi Ollila <tomi.ollila@iki.fi>
Subject: Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general
	refactoring
Message-ID: <20140204161142.GS14197@odin.tremily.us>
References: <cover.1391424512.git.wking@tremily.us>
	<m2ha8frjgg.fsf@guru.guru-group.fi>
	<20140204005331.GQ14197@odin.tremily.us>
	<m2fvnzp3uh.fsf@guru.guru-group.fi>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="lFtSGh4BwI5kpuz7"
Content-Disposition: inline
In-Reply-To: <m2fvnzp3uh.fsf@guru.guru-group.fi>
OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8;
	url=http://tremily.us/pubkey.txt
User-Agent: Mutt/1.5.22 (2013-10-16)
Cc: notmuch@notmuchmail.org
Precedence: list


--lFtSGh4BwI5kpuz7
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, "W. Trevor King" wrote:
> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
> >>   File "devel/nmbug/nmbug-status", line 197, in _write_threads
> >>     ).format(**message_display_data))
> >>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
> >>     data, consumed =3D self.encode(object, self.errors)
> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
> >>   position 176: ordinal not in range(128)
> >
> > Hmm.  __future__'s unicode_literals should be giving us a Unicode
> > target, so I'm not sure why we'd have trouble injecting Unicode.  This
> > works fine for me on Python 2.7 and 3.3.  Maybe you just have a funky
> > encoding?  =E2=80=A6
>=20
> LANG=3Den_US.UTF-8
> all other LC_* variables en_US.UTF-8 except
> LC_TIME=3Den_GB.utf8
> LC_ALL empty (naturally)
>=20
> python -c 'import locale; print(locale.getpreferredencoding())'
> UTF-8
> python -c 'import sys; print(sys.getdefaultencoding())'
> ascii

That's very strange.  On Python 2.6.9, with the same encodings:

  >>> from __future__ import unicode_literals
  >>> import codecs
  >>> import locale
  >>> import sys
  >>> print(locale.getpreferredencoding())  # same as yours
  UTF-8
  >>> print(sys.getdefaultencoding())  # same as yours
  ascii
  >>> _ENCODING =3D locale.getpreferredencoding() or sys.getdefaultencoding=
()
  >>> print(_ENCODING)  # double-check default encodings
  UTF-8
  >>> byte_stream =3D sys.stdout  # copied from Page.write
  >>> stream =3D codecs.getwriter(encoding=3D_ENCODING)(stream=3Dbyte_strea=
m)
  >>> data =3D {'from': '\u017b'}  # fake the troublesome data
  >>> print(type(data['from']))  # double-check unicode_literals
  <type 'unicode'>
  >>> string =3D '  <td>{from}</td>\n'.format(**data)
  >>> stream.write(string)
    <td>=C5=BB</td>

It looks like you'll have the same _ENCODING as I do (UTF-8).  That
means your stream should be wrapped in a UTF-8 StreamWriter, so I
don't understand why it's converting to ASCII.  Can you run through
the above on your troublesome machine and confirm that stream.write()
is still raising the exception?  If it doesn't work, can you just
paste that whole run in your next email?

Thanks,
Trevor

--=20
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

--lFtSGh4BwI5kpuz7
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJS8RE7AAoJEKKfehoaNkbtIp8P+wYnJZsXqoGbGtfxvibDe2K1
xsGvn2gcOk+NgQU6O1v58l/07jetldl0Q2uMVvGgXv7us3VS6ZqttJGfYdTXK2XH
phJXsrB6C7O68Tff2kHXTwTdtJih5Ly7TiKrQdl+jVAwBZ5LNtHb0x/I0CVe5JpU
Wu94fPePfufa+SlWSOGOnJqNMY4iXFUAkXNWMogTa3luVkUhShqFvSyacCZXr0ch
rKlSMk82NgVk/zo2nNsrq9iEOiEtQCShkA72hU1tW3rS1W1xw2yXAa01GGj/yk4Y
glrt0A8kQNP4/J8MGv+/19AXO1y4B/HWlOqWNJ/mJ/3pRtkCb4xJ4+5cwaBNZw3P
3bXsKLkoTTReee2lAfpqBedP/bhBUleY2CzTihY49NeoCnOPNGDzj0NEoiaw0Krx
cyI7pLPEQ2TxOEeewTqvgzUnBGNwuJdF4a13uYqMn+I9+8psnwYgCku0J6gjra8t
9zsFYid3bB3uEfyoMJ1Jib3ret21/C1QQFbJnkeS1DxjJkGRmoIKaVCUvrE8mtTQ
8lSoly8IG17tgOvANYzZkHSMM3cVFqG9Qd29xQp/zE7VRVi8ipi/fji8xHvyOWcf
TYB00ToBpZCus8N6ggC229bU1QhnvVVDJmIpCzeMHKsqSlqClR1xqEZvEGQJktI8
ZfUAM6Ca4l+VqtTjE0Cu
=lj0k
-----END PGP SIGNATURE-----

--lFtSGh4BwI5kpuz7--