From: Jesse Rosenthal <jrosenthal@jhu.edu>
To: Notmuch developer list <notmuch@notmuchmail.org>
Subject: [PATCH] Modularize test suite
Date: Fri, 30 Apr 2010 14:36:56 -0400
Message-ID: <87sk6cdbxz.fsf@jhu.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Precedence: list

Responding to this email is a patch to modularize the test suite. It
follows a relatively simple model, but it does seem like it will offer
some degree of flexibility. Plus, no relicensing is necessary.

In short, tests are put in subdirs of test/tests, e.g.:

     test/tests/notmuch-new/
     test/tests/notmuch-search/
     test/tests/json/
     ...

"notmuch-test" (which is the same as "notmuch-test all") should work as
usual. But if you call "notmuch-test json", it will only the json tests.

It will do this by running all of the files in the json subdir that have
an extension "*.test" or "*.setup" (all other files will be ignored). It
runs them in numerical order, which you can set by a numerical
prefix. So, for example, in test/tests/notmuch-reply:

         00001-notmuch-reply.setup
         00100-basic-reply.test          
         00200-multiple-recipients.test  
         00300-reply-with-cc.test        
         00400-reply-from-alternate-address.test
         00500-support-for-reply-to.test
         00600-unmunging-reply-to.test
         99999-notmuch-reply.setup

The basic numbering system is 00100, 00200, etc. to both allow a lot of
space to fit things in between, and to let the collection of tests
grow. 

00001-$(basename $(pwd)).setup and 99999-$(basename $(pwd)).setup are
reserved for setting up and cleaning up. This allows us to call
individual sets of tests without calling the entire suite in order to
get the environment that the tests expect.

Note that if you add a new subdir collection of tests, you have to add
it to the $TEST_COLLECTIONS variable in notmuch-test. It doesn't run all
subdirs automatically, in case you want to test something out, or have a
subdir that only works in certain cases (imagine future tests of remote
setups).

Sorry the patch is so large: there was a lot of back-and-forth, and it
was hard to break the patches down into coherent steps. Carl, if you'd
prefer, I could send you a more granular, but confusing, series of
patches.

Best,
Jesse