From: W. Trevor King Date: Sun, 14 Feb 2016 06:31:32 +0000 (+1600) Subject: Re: problems with nmbug and empty prefix (UnicodeWarning and broken pipe) X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=a71c5b36bf63c1eacd596c43eb546155051f0f3c;p=notmuch-archives.git Re: problems with nmbug and empty prefix (UnicodeWarning and broken pipe) --- diff --git a/f7/96f9da6feeff8903e275cddb3ec3d635a73881 b/f7/96f9da6feeff8903e275cddb3ec3d635a73881 new file mode 100644 index 000000000..66307eccf --- /dev/null +++ b/f7/96f9da6feeff8903e275cddb3ec3d635a73881 @@ -0,0 +1,198 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by arlo.cworth.org (Postfix) with ESMTP id 6E0206DE0AF8 + for ; Sat, 13 Feb 2016 22:31:39 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at cworth.org +X-Spam-Flag: NO +X-Spam-Score: 0.007 +X-Spam-Level: +X-Spam-Status: No, score=0.007 tagged_above=-999 required=5 tests=[AWL=0.108, + DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled +Received: from arlo.cworth.org ([127.0.0.1]) + by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id jHIQUFT-wLaf for ; + Sat, 13 Feb 2016 22:31:36 -0800 (PST) +Received: from resqmta-po-10v.sys.comcast.net (resqmta-po-10v.sys.comcast.net + [96.114.154.169]) + by arlo.cworth.org (Postfix) with ESMTPS id 392CB6DE0A9A + for ; Sat, 13 Feb 2016 22:31:35 -0800 (PST) +Received: from resomta-po-19v.sys.comcast.net ([96.114.154.243]) + by resqmta-po-10v.sys.comcast.net with comcast + id J6XV1s0025FMDhs016XaqT; Sun, 14 Feb 2016 06:31:34 +0000 +Received: from mail.tremily.us ([73.221.72.168]) + by resomta-po-19v.sys.comcast.net with comcast + id J6XY1s00J3dr3C9016XZ8s; Sun, 14 Feb 2016 06:31:33 +0000 +Received: by mail.tremily.us (Postfix, from userid 1000) + id 7018C1BB1A75; Sat, 13 Feb 2016 22:31:32 -0800 (PST) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tremily.us; s=odin; + t=1455431492; bh=doMnITzmjEm/nfzTRvyZujrx5mV36yKmPCZQnXGrkss=; + h=Date:From:To:Cc:Subject:References:In-Reply-To; + b=kF6iJAdjcQZJNAK/OlXWMxyBDmBpvLuEO1iVr4S+ldRN5+2BJtiVoFFscy0NfeJAh + 7t1gdCKy2Llv2oUdvto6uohoafvUxg4C+3Tg4TOhmspysMMa5eBk+wERZSfbbllmMP + 5No8kbxW7dJg41qh6uBUo8rnUDeTZNvMbDv1AUTU= +Date: Sat, 13 Feb 2016 22:31:32 -0800 +From: "W. Trevor King" +To: David Bremner +Cc: notmuch@notmuchmail.org +Subject: Re: problems with nmbug and empty prefix (UnicodeWarning and broken + pipe) +Message-ID: <20160214063132.GD4265@odin.tremily.us> +References: <87oabko293.fsf@zancas.localnet> + <20160213223357.GC4265@odin.tremily.us> + <87ziv4813v.fsf@zancas.localnet> +MIME-Version: 1.0 +Content-Type: multipart/signed; micalg=pgp-sha1; + protocol="application/pgp-signature"; boundary="bMDD5T6Tp9Yu3dvb" +Content-Disposition: inline +In-Reply-To: <87ziv4813v.fsf@zancas.localnet> +OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8; + url=http://tremily.us/pubkey.txt +User-Agent: Mutt/1.5.23 (2014-03-12) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; + s=q20140121; t=1455431494; + bh=MDLTWBSn1+hD9ci3RkDBCXFTpKUbbeQrMCqIoP35rPQ=; + h=Received:Received:Received:Date:From:To:Subject:Message-ID: + MIME-Version:Content-Type; + b=mNvKfayACyMkwVoXxWH+zVhLzHNKowXBuMQGYImobuSr/78R7Ry48Oy6gWY/a5kxW + DDKHc9ALCbmwA+i+t/i3wOzj6qNUiD9ntvk9F0FBn07lG/ujf56gnB7/ZInWSGVeai + Y09Be4xqDNUeRGu0JApmiTbXS0phTFbr2w7n5rOn3cyjJY5qsL/dC7opgJa22xWLXb + 5vYUY/EsfcztfIZLRC+7/DV+k87yzGY1RGEVhvPei6OmAVeXs/hGJuFAS43aT+76Zs + J6ARECHtk7ummoYNt6k36aZv4QeRd0uRnWHdBKaE4Qv5sCT+q2UNSy5dkMP4MmVDng + E6Cns7RLqNwIQ== +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.20 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 14 Feb 2016 06:31:39 -0000 + + +--bMDD5T6Tp9Yu3dvb +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline +Content-Transfer-Encoding: quoted-printable + +On Sat, Feb 13, 2016 at 10:41:40PM -0400, David Bremner wrote: +> Traceback (most recent call last): +> File "/home/bremner/.config/scripts/nmbug.real", line 834, in +> args.func(**kwargs) +> File "/home/bremner/.config/scripts/nmbug.real", line 324, in commit +> status =3D get_status() +> File "/home/bremner/.config/scripts/nmbug.real", line 581, in get_status +> index =3D _index_tags() +> File "/home/bremner/.config/scripts/nmbug.real", line 621, in _index_ta= +gs +> git.stdin.write(line) + +This traceback is pointing at what should be a stream write, so I +don't see how urllib is involved there at all. I guess this traceback +ends up in the =E2=80=9CBroken pipe=E2=80=9D message from your original pos= +t? + +Dropping some debugging prints into the: + + for line in notmuch.stdout: + +block will likely get us close enough to figure out which line in the +=E2=80=98notmuch dump =E2=80=A6=E2=80=99 output causing the problem. + +> > We only call =E2=80=98notmuch dump =E2=80=A6=E2=80=99 from _index_tags,= + where dump's stdout is +> > tweaked and fed into =E2=80=98git update-index =E2=80=A6=E2=80=99. You= +r urllib UnicodeWarning +> > suggests the issue lies in: +> > +> > tags =3D [ +> > _unquote(tag[len(prefix):]) +> > for tag in tags_string.split() +> > if tag.startswith(prefix)] +>=20 +> Looking at the source for urllib, that line is actually in quote, +> which is called only from _hex_quote + +And we call _hex_quote from _index_tags_for_message, which is right +before the git.stdin.write line from your traceback. So its certainly +possible that we're feeding _hex_quote something it can't handle in +Python 2. If I could reproduce this locally, I'd probably drop a +debugging print in there as well: + + for tag in tags: + _LOG.debug('building a quoted path for {!r} / {!r}'.format(id, tag)) + path =3D 'tags/{id}/{tag}'.format( + id=3D_hex_quote(string=3Did), tag=3D_hex_quote(string=3Dtag)) + yield '{mode} {hash}\t{path}\n'.format(mode=3Dmode, hash=3Dhash, path= +=3Dpath) + +> Unfortunately despite my best efforts with filterwarnings, I +> couldn't figure out how to get a stack trace for that +> UnicodeWarning. + +I haven't spent much time with filterwarnings. My guess is that: + + $ python -W error ./nmbug --log-level debug commit + +will turn it into a raised exception [1]. But you may have tried +that, and it may not have worked for some reason :p. + +If dropping debugging prints into the relevant code sections doesn't +turn up the problem, =E2=80=98strace -o /tmp/trace -f nmbug --log-level deb= +ug +commit=E2=80=99 will likely capture enough of the data moving between +processes for us to figure out what nmbug is choking on. + +Another alternative would be to check your list of censored tags for +anything that looks like it might contain Unicode-issue-triggering +characters. What is your locale? Do you have any tags with non-ASCII +characters? You should be able to isolate this problem by iterating +through all your tags: + + $ for TAG in + > do + > echo "${TAG}" + > NMBPREFIX=3D"${TAG%?}" nmbug commit + > done + +and see which one acts up. + +Cheers, +Trevor + +[1]: https://docs.python.org/2/library/warnings.html#warning-filter + +--=20 +This email may be signed or encrypted with GnuPG (http://www.gnupg.org). +For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy + +--bMDD5T6Tp9Yu3dvb +Content-Type: application/pgp-signature; name="signature.asc" +Content-Description: OpenPGP digital signature + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v2 + +iQIcBAEBAgAGBQJWwB9BAAoJEAPqygegUbGsJ5wQAKiq2gt5YYHb02ZW4gT3nfv6 +2QdO1FlUy9XhPhzwPKdK8NBeIZIhzI0Pu95/kXRauQxPgveZlgaDeBdEPYS9G1hf +yuwwfXKclqKF2dNnieJ7EkdZ2KqPkgOpVWjoffONzGGhNBMUIzFR1pwtoJwxHSwS +XE1Xhp91FBOQ2EbxTCHI1mgwQI49YVhZffIKkg2hrmIqieg/VgrU0fpEGVRCf6zr +yfk1sbUKRxG/fC+B5lLJHhbu+A3a2InJtlgk+IyXXv8UQcFIBA34sxYqWI6Jxujl +Jp5LM49DeiMmaL12PnDeCYJh6MfZdnVqq/LkeAzhVmVSNfRZ9DJUIr4zKlzY3lRU +Bxge7yaEH/ZboQMOM1y06UqHcIBY+05OZNjCep+I/4dlQlYmDPCBR7MY5WdACVl5 +Je5F3Fr5ZZwyNT5K/6IhgWCT1d0IWIh8m0GY6VfsFasynzZLSEyOleU7SXFiQAjb +vpk5jWsUIOGlWEwXnv+h7kdwRvY9Q+w8h8wON1RAFI+hEvPV2XFq2xqjvXqebsRd +UdsFRxyQn7AWU3BuKGyo6p5RF1k3W+miCqnDR5h5IZSFQHTDjRUKcfP7rz0ep6CM +so4Y23NTuQbRK20POiE9Ig/3MeCwWYfqs2uMr8/dZ5jg0UCk4oXU/XLwnx50kPxx +injrOxzmqVKm4C9R1DtY +=1g4t +-----END PGP SIGNATURE----- + +--bMDD5T6Tp9Yu3dvb--