Sane use of test_expect_failure
authorJunio C Hamano <gitster@pobox.com>
Fri, 1 Feb 2008 09:50:53 +0000 (01:50 -0800)
committerJunio C Hamano <gitster@pobox.com>
Sat, 2 Feb 2008 04:49:34 +0000 (20:49 -0800)
commit41ac414ea2bef81af94474cbef25a38868b4788e
treee9c598e65753ab473eefc6fcc5899714d8085a2f
parent6ce8e44a1eeaa07325f1304f6f392f35f54d29c7
Sane use of test_expect_failure

Originally, test_expect_failure was designed to be the opposite
of test_expect_success, but this was a bad decision.  Most tests
run a series of commands that leads to the single command that
needs to be tested, like this:

    test_expect_{success,failure} 'test title' '
setup1 &&
        setup2 &&
        setup3 &&
        what is to be tested
    '

And expecting a failure exit from the whole sequence misses the
point of writing tests.  Your setup$N that are supposed to
succeed may have failed without even reaching what you are
trying to test.  The only valid use of test_expect_failure is to
check a trivial single command that is expected to fail, which
is a minority in tests of Porcelain-ish commands.

This large-ish patch rewrites all uses of test_expect_failure to
use test_expect_success and rewrites the condition of what is
tested, like this:

    test_expect_success 'test title' '
setup1 &&
        setup2 &&
        setup3 &&
        ! this command should fail
    '

test_expect_failure is redefined to serve as a reminder that
that test *should* succeed but due to a known breakage in git it
currently does not pass.  So if git-foo command should create a
file 'bar' but you discovered a bug that it doesn't, you can
write a test like this:

    test_expect_failure 'git-foo should create bar' '
        rm -f bar &&
        git foo &&
        test -f bar
    '

This construct acts similar to test_expect_success, but instead
of reporting "ok/FAIL" like test_expect_success does, the
outcome is reported as "FIXED/still broken".

Signed-off-by: Junio C Hamano <gitster@pobox.com>
48 files changed:
t/README
t/t0000-basic.sh
t/t0030-stripspace.sh
t/t0040-parse-options.sh
t/t1000-read-tree-m-3way.sh
t/t1200-tutorial.sh
t/t1300-repo-config.sh
t/t1302-repo-version.sh
t/t1400-update-ref.sh
t/t2000-checkout-cache-clash.sh
t/t2002-checkout-cache-u.sh
t/t2008-checkout-subdir.sh
t/t2100-update-cache-badpath.sh
t/t3020-ls-files-error-unmatch.sh
t/t3200-branch.sh
t/t3210-pack-refs.sh
t/t3400-rebase.sh
t/t3403-rebase-skip.sh
t/t3600-rm.sh
t/t4103-apply-binary.sh
t/t4113-apply-ending.sh
t/t5300-pack-object.sh
t/t5302-pack-index.sh
t/t5401-update-hooks.sh
t/t5402-post-merge-hook.sh
t/t5500-fetch-pack.sh
t/t5510-fetch.sh
t/t5530-upload-pack-error.sh
t/t5600-clone-fail-cleanup.sh
t/t5710-info-alternate.sh
t/t6023-merge-file.sh
t/t6024-recursive-merge.sh
t/t6025-merge-symlinks.sh
t/t6101-rev-parse-parents.sh
t/t6300-for-each-ref.sh
t/t7001-mv.sh
t/t7002-grep.sh
t/t7004-tag.sh
t/t7101-reset.sh
t/t7501-commit.sh
t/t7503-pre-commit-hook.sh
t/t7504-commit-msg-hook.sh
t/t9100-git-svn-basic.sh
t/t9106-git-svn-commit-diff-clobber.sh
t/t9106-git-svn-dcommit-clobber-series.sh
t/t9300-fast-import.sh
t/t9400-git-cvsserver-server.sh
t/test-lib.sh