Date: Sat, 13 Feb 2016 22:31:32 -0800
From: "W. Trevor King" <wking@tremily.us>
To: David Bremner <david@tethera.net>
Cc: notmuch@notmuchmail.org
Subject: Re: problems with nmbug and empty prefix (UnicodeWarning and broken
 pipe)
Message-ID: <20160214063132.GD4265@odin.tremily.us>
References: <87oabko293.fsf@zancas.localnet>
 <20160213223357.GC4265@odin.tremily.us>
 <87ziv4813v.fsf@zancas.localnet>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
 protocol="application/pgp-signature"; boundary="bMDD5T6Tp9Yu3dvb"
Content-Disposition: inline
In-Reply-To: <87ziv4813v.fsf@zancas.localnet>
OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8;
 url=http://tremily.us/pubkey.txt
User-Agent: Mutt/1.5.23 (2014-03-12)
Precedence: list


--bMDD5T6Tp9Yu3dvb
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Feb 13, 2016 at 10:41:40PM -0400, David Bremner wrote:
> Traceback (most recent call last):
>   File "/home/bremner/.config/scripts/nmbug.real", line 834, in <module>
>     args.func(**kwargs)
>   File "/home/bremner/.config/scripts/nmbug.real", line 324, in commit
>     status =3D get_status()
>   File "/home/bremner/.config/scripts/nmbug.real", line 581, in get_status
>     index =3D _index_tags()
>   File "/home/bremner/.config/scripts/nmbug.real", line 621, in _index_ta=
gs
>     git.stdin.write(line)

This traceback is pointing at what should be a stream write, so I
don't see how urllib is involved there at all.  I guess this traceback
ends up in the =E2=80=9CBroken pipe=E2=80=9D message from your original pos=
t?

Dropping some debugging prints into the:

  for line in notmuch.stdout:

block will likely get us close enough to figure out which line in the
=E2=80=98notmuch dump =E2=80=A6=E2=80=99 output causing the problem.

> > We only call =E2=80=98notmuch dump =E2=80=A6=E2=80=99 from _index_tags,=
 where dump's stdout is
> > tweaked and fed into =E2=80=98git update-index =E2=80=A6=E2=80=99.  You=
r urllib UnicodeWarning
> > suggests the issue lies in:
> >
> >   tags =3D [
> >       _unquote(tag[len(prefix):])
> >       for tag in tags_string.split()
> >       if tag.startswith(prefix)]
>=20
> Looking at the source for urllib, that line is actually in quote,
> which is called only from _hex_quote

And we call _hex_quote from _index_tags_for_message, which is right
before the git.stdin.write line from your traceback.  So its certainly
possible that we're feeding _hex_quote something it can't handle in
Python 2.  If I could reproduce this locally, I'd probably drop a
debugging print in there as well:

  for tag in tags:
      _LOG.debug('building a quoted path for {!r} / {!r}'.format(id, tag))
      path =3D 'tags/{id}/{tag}'.format(
          id=3D_hex_quote(string=3Did), tag=3D_hex_quote(string=3Dtag))
      yield '{mode} {hash}\t{path}\n'.format(mode=3Dmode, hash=3Dhash, path=
=3Dpath)

> Unfortunately despite my best efforts with filterwarnings, I
> couldn't figure out how to get a stack trace for that
> UnicodeWarning.

I haven't spent much time with filterwarnings.  My guess is that:

  $ python -W error ./nmbug --log-level debug commit

will turn it into a raised exception [1].  But you may have tried
that, and it may not have worked for some reason :p.

If dropping debugging prints into the relevant code sections doesn't
turn up the problem, =E2=80=98strace -o /tmp/trace -f nmbug --log-level deb=
ug
commit=E2=80=99 will likely capture enough of the data moving between
processes for us to figure out what nmbug is choking on.

Another alternative would be to check your list of censored tags for
anything that looks like it might contain Unicode-issue-triggering
characters.  What is your locale?  Do you have any tags with non-ASCII
characters?  You should be able to isolate this problem by iterating
through all your tags:

  $ for TAG in <censored>
  > do
  >   echo "${TAG}"
  >   NMBPREFIX=3D"${TAG%?}" nmbug commit
  > done

and see which one acts up.

Cheers,
Trevor

[1]: https://docs.python.org/2/library/warnings.html#warning-filter

--=20
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

--bMDD5T6Tp9Yu3dvb
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJWwB9BAAoJEAPqygegUbGsJ5wQAKiq2gt5YYHb02ZW4gT3nfv6
2QdO1FlUy9XhPhzwPKdK8NBeIZIhzI0Pu95/kXRauQxPgveZlgaDeBdEPYS9G1hf
yuwwfXKclqKF2dNnieJ7EkdZ2KqPkgOpVWjoffONzGGhNBMUIzFR1pwtoJwxHSwS
XE1Xhp91FBOQ2EbxTCHI1mgwQI49YVhZffIKkg2hrmIqieg/VgrU0fpEGVRCf6zr
yfk1sbUKRxG/fC+B5lLJHhbu+A3a2InJtlgk+IyXXv8UQcFIBA34sxYqWI6Jxujl
Jp5LM49DeiMmaL12PnDeCYJh6MfZdnVqq/LkeAzhVmVSNfRZ9DJUIr4zKlzY3lRU
Bxge7yaEH/ZboQMOM1y06UqHcIBY+05OZNjCep+I/4dlQlYmDPCBR7MY5WdACVl5
Je5F3Fr5ZZwyNT5K/6IhgWCT1d0IWIh8m0GY6VfsFasynzZLSEyOleU7SXFiQAjb
vpk5jWsUIOGlWEwXnv+h7kdwRvY9Q+w8h8wON1RAFI+hEvPV2XFq2xqjvXqebsRd
UdsFRxyQn7AWU3BuKGyo6p5RF1k3W+miCqnDR5h5IZSFQHTDjRUKcfP7rz0ep6CM
so4Y23NTuQbRK20POiE9Ig/3MeCwWYfqs2uMr8/dZ5jg0UCk4oXU/XLwnx50kPxx
injrOxzmqVKm4C9R1DtY
=1g4t
-----END PGP SIGNATURE-----

--bMDD5T6Tp9Yu3dvb--