From: Carl Worth <cworth@cworth.org>
To: Michal Sojka <sojkam1@fel.cvut.cz>, notmuch@notmuchmail.org
Subject: Re: [PATCH] test: Better handling of stdout and stderr
In-Reply-To: <1289346035-24904-1-git-send-email-sojkam1@fel.cvut.cz>
References: <1289346035-24904-1-git-send-email-sojkam1@fel.cvut.cz>
User-Agent: Notmuch/0.4 (http://notmuchmail.org) Emacs/23.2.1
	(i486-pc-linux-gnu)
Date: Wed, 10 Nov 2010 13:15:45 -0800
Message-ID: <87k4kkq3r2.fsf@yoom.home.cworth.org>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"
Precedence: list

--=-=-=
Content-Transfer-Encoding: quoted-printable

On Wed, 10 Nov 2010 00:40:35 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrot=
e:
> Git-style tests suppress stdout and stderr unless -v is given.
> cworth-style tests (created by test_begin_subtest, test_expect_equal)
> do not have this behavior so implement it the same.
>=20
> Additionally, for both test styles, the test-lib.sh is changed so that
> the content of suppressed stdout and stderr is shown in case of failed
> test.
>=20
> Finally a test for this functionality is added to basic tests

This looks like some great stuff, Michal! I especially like the way you
are testing the test-suite framework itself[*].

I did have one local commit to test-lib.sh that broke this slightly, (I
changed it so that you will now get intermediate files like
test-verbose.4.out rather than test.4.out). That would have been easy
enough to fix, but I also noticed output like the following from "make
test":

...
	 PASS   Search by to: (address)
	 PASS   Search by to: (name)
	 PASS   Search by subject: (phrase)
	json: Testing --format=3Djson output
	 PASS   Show message: json
	 PASS   Search message: json
	 PASS   Search by subject (utf-8):
	 PASS   Show message: json, utf-8
	 PASS   Search message: json, utf-8
=20=20=20=20=20=20=20=20=20
	thread-naming: Testing naming of threads with changing subject
	raw: Testing notmuch show --format=3Draw
	 PASS   Generate some messages
	 PASS   Attempt to show multiple raw messages
...

At first I thought the change just caused some newlines to be dropped,
(such as the missing newline just before the "json" line). But then I
noticed that all output from all of the thread-naming tests is entirely
missing.

Do you want to look into what's going on there?

Meanwhile, I've just now pushed out my little test-lib.sh change, so you
can update the patch for that as well.

Thanks,

=2DCarl

[*] The current git test suite tries to do some things like this, but
not as cleverly. So it actually reports a FAIL and a BROKEN test when in
fact it's only trying to exercise the parts of the test suite that
report FAIL and BROKEN. I found that objectionable, so simply removed
those tests. But calling to a subordinate test that would report FAIL
and then reporting PASS after checking that would be just fine. That's a
change we could implement and even contribute upstream to the git
project itself.

=2D-=20
carl.d.worth@intel.com

--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iD8DBQFM2wuB6JDdNq8qSWgRAv3vAJ9NeaK0icfj9/+9GtVLIBO1jfDEZgCgmI6W
fA/F3EcxfByPmF6toktPmF4=
=mrFS
-----END PGP SIGNATURE-----
--=-=-=--