From: Mark Walters <markwalters1009@gmail.com>
To: notmuch@notmuchmail.org
Subject: Tests
User-Agent: Notmuch/0.13.2+61~gf708609 (http://notmuchmail.org) Emacs/23.4.1
	(x86_64-pc-linux-gnu)
Date: Tue, 10 Jul 2012 17:51:48 +0100
Message-ID: <87ipdv8spn.fsf@qmul.ac.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Precedence: list


I have been thinking a little bit about the current situation with
regards to tests. There are quite a lot of tests in the review queue
that have been there for quite some time without much interest, but I do
think we are rather short of tests. (*)

I wonder if we could have a sort of staging area for tests where they
could roughly go in without review and would only be run by something
like make stage-test or make all-tests or something.  The hope is that
this would encourage more tests.

The idea would be not that a patch author would have to make sure they
all pass, but if they previously passed and no longer do then the author
would know *something* about the output had changed.

One example is the emacs elide test by Pieter
id:"1329684990-12504-3-git-send-email-pieter@praet.org" which does still
pass after my substantial change to the way elide is done (and is not
currently covered in the test suite)

Of course if someone does review one of these staging tests then they
can be moved into the real tests.

Best wishes

Mark

(*) for example the structured output patch accidentally changed the
output of notmuch search --output=threads --format=json <search> but
that was not caught by the tests.