From: Carl Worth <cworth@cworth.org>
To: "Jeffrey C. Ollie" <jeff@ocjtech.us>, Not Much Mail
	<notmuch@notmuchmail.org>
In-Reply-To: <1259424123-26285-1-git-send-email-jeff@ocjtech.us>
References: <1259424123-26285-1-git-send-email-jeff@ocjtech.us>
Date: Sun, 10 Jan 2010 09:54:25 -0800
Message-ID: <87my0llub2.fsf@yoom.home.cworth.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Subject: Re: [notmuch] [PATCH] Add the beginnings of a test suite.
Precedence: list

--=-=-=
Content-Type: multipart/signed; boundary="==-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"

--==-=-=

On Sat, 28 Nov 2009 10:02:03 -0600, "Jeffrey C. Ollie" <jeff@ocjtech.us> wrote:
> This is the beginning of a test suite.  It uses the Check[1] unit
> testing framework to handle the testing.  There's one basic test of
> the SHA1 routines, obviously many more will need to be added.

Hi Jeffrey,

Thanks very much for this contribution. I'm always highly in favor of
improving our testing infrastructure. And a good test suite was on of
the first items I put on the notmuch/TODO list.

However, it's not clear to me that this kind of C-based unit testing
will be able to easily support much of what I want from testing for
notmuch. It seems that most of the bugs I encounter, or features that I
want to test, require a lot of state to be setup that would be really
painful to do in C code.

For example, a recent bug I hit was:

	After moving an (old) existing file into the mail store,
	"notmuch new" won't pick it up.

And I don't see an easy way to test something like that from a typical
"check" test. Meanwhile, I can't see a lot of use from the kinds of
things that *could* be checked from a check test. We could easily check
little things like that after calling notmuch_message_add_tag that
notmuch_message_get_tags has the same tag in its list. But if we have to
do higher-level testing anyway, (for cases like the bug above), then
it's easy to cover fine-grained stuff like this there as well.

For the recent rename-support work I did I wrote the attached
notmuch-test script. It does the kind of high-level state setup I
described above, (and actually contains a test for precisely the bug I
described above).

I haven't added this to the repository yet since it needs a bunch of
work before being usable. Here are some things it needs:

  * Automated verification that tests are working, (currently I'm just
    manually verifying that the results are as expected---obviously
    that's not practical in the long term).

  * Modularization so that each test can be maintained as a small,
    independent snippet.

  * The ability to run an individual test in isolation without running
    the whole suite.

  * A fix in notmuch to get rid of the stupid sleep calls that are
    slowing down the current notmuch-test. (One approach would be to
    stop using mtime altogether. That might slow things down
    unacceptably in general use, but might be just fine for the test
    suite where the mail store can generally be quite tiny. Another
    approach would be to use a filesystem like ext4 with sub-second
    mtime support).

Anyway, just wanted to share some thoughts and some preliminary code for
a notmuch test suite.

-Carl


--==-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iD8DBQFLShRR6JDdNq8qSWgRAsb/AJ98eYj50/tLpFvEeusQp64uPRws+wCfThoc
4IUXgxTd86itranVdy3iazU=
=O/dO
-----END PGP SIGNATURE-----
--==-=-=--

--=-=-=
Content-Disposition: inline; filename=notmuch-test
Content-Description: Lame prototype of a notmuch test suite

#!/bin/sh
set -e

find_notmuch_binary ()
{
    dir=$1

    while [ -n "$dir" ]; do
	bin=$dir/notmuch
	if [ -x $bin ]; then
	    echo $bin
	    return
	fi
	dir=$(dirname $dir)
	if [ "$dir" = "/" ]; then
	    break
	fi
    done

    echo notmuch
}

# Generate a new message in the mail directory, with
# a unique message ID and subject.
#
# The filename of the message generated is available as
# $gen_msg_filename
gen_msg_cnt=0
gen_msg_filename=""
generate_message ()
{
    gen_msg_cnt=$((gen_msg_cnt + 1))
    gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)

    if [ "$#" = "0" ]; then
	gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
    else
	gen_msg_filename="${MAIL_DIR}/$1/$gen_msg_name"
	mkdir -p $(dirname $gen_msg_filename)
    fi

cat <<EOF >$gen_msg_filename
From: Notmuch Test Suite <test_suite@notmuchmail.org>
To: Notmuch Test Suite <test_suite@notmuchmail.org>
Message-Id: <msg-${gen_msg_cnt}@notmuch-test-suite>
Subject: Test message ${gen_msg_filename}
Date: Tue, 05 Jan 2010 15:43:57 -0800

This is just a test message at ${gen_msg_filename}
EOF
}

do_sleep ()
{
    sleep 1
}

TEST_DIR=$(pwd)/test.$$
MAIL_DIR=${TEST_DIR}/mail
export NOTMUCH_CONFIG=${TEST_DIR}/notmuch-config
NOTMUCH=$(find_notmuch_binary $(pwd))

rm -rf ${TEST_DIR}
mkdir ${TEST_DIR}
cd ${TEST_DIR}

mkdir ${MAIL_DIR}

cat <<EOF > ${NOTMUCH_CONFIG}
[database]
path=${MAIL_DIR}

[user]
name=Notmuch Test Suite
primary_email=test_suite@notmuchmail.org
EOF

echo "### Testing \"notmuch new\" with no messages"
$NOTMUCH new

echo "### Testing \"notmuch new\" with 1 new message"
do_sleep
generate_message
$NOTMUCH new

echo "### Testing \"notmuch new\" with 2 new messages"
do_sleep
generate_message
generate_message
$NOTMUCH new

echo "### Testing \"notmuch new\" with no new messages (and a non-empty database)"

$NOTMUCH new

echo "### Testing \"notmuch new\" with two new directories (one mail)"
rm -rf ${MAIL_DIR}/* ${MAIL_DIR}/.notmuch
mkdir ${MAIL_DIR}/def
mkdir ${MAIL_DIR}/ghi
generate_message def

$NOTMUCH new

echo "### Testing \"notmuch new\" with two new directories (one mail)---opposite inode order"

rm -rf ${MAIL_DIR}/.notmuch
mv ${MAIL_DIR}/ghi ${MAIL_DIR}/abc
rm ${MAIL_DIR}/def/*
generate_message abc

$NOTMUCH new

echo "### Testing \"notmuch new\" with 1 old message moved into the mail store"
rm -rf ${MAIL_DIR}/* ${MAIL_DIR}/.notmuch
generate_message
tmp_msg_filename=tmp/$gen_msg_filename
mkdir -p $(dirname $tmp_msg_filename)
mv $gen_msg_filename $tmp_msg_filename
do_sleep
$NOTMUCH new > /dev/null
do_sleep
mv $tmp_msg_filename $gen_msg_filename
$NOTMUCH new

echo "### Testing \"notmuch new\" with 1 renamed message"

do_sleep
generate_message
$NOTMUCH new > /dev/null
do_sleep
mv $gen_msg_filename ${gen_msg_filename}-renamed
$NOTMUCH new

echo "### Testing \"notmuch new\" with 1 deleted message"

do_sleep
rm ${gen_msg_filename}-renamed
$NOTMUCH new

echo "### Testing \"notmuch new\" with a new directory with 3 messages"

do_sleep
generate_message dir
generate_message dir
generate_message dir

$NOTMUCH new

echo "### Testing \"notmuch new\" with a renamed directory of 3 messages"

do_sleep
mv ${MAIL_DIR}/dir ${MAIL_DIR}/dir-renamed

$NOTMUCH new

echo "### Testing \"notmuch new\" with a deleted directory of 3 messages"

do_sleep
rm -rf ${MAIL_DIR}/dir-renamed

$NOTMUCH new

echo "### Testing \"notmuch new\" with a new directory with 3 messages (tail of list)"

do_sleep
generate_message zzz
generate_message zzz
generate_message zzz

$NOTMUCH new

echo "### Testing \"notmuch new\" with a deleted directory of 3 messages (tail of list)"

do_sleep
rm -rf ${MAIL_DIR}/zzz

$NOTMUCH new

echo "### Testing \"notmuch new\" with a symlink to an external directory of 1 message"

rm -rf ${MAIL_DIR}/.notmuch
mv ${MAIL_DIR} ${TEST_DIR}/actual_maildir

mkdir ${MAIL_DIR}
ln -s ${TEST_DIR}/actual_maildir ${MAIL_DIR}/symlink

$NOTMUCH new

echo "### Testing \"notmuch new\" with a symlink to an external file"
do_sleep
generate_message
external_msg_filename=${TEST_DIR}/external/$(basename $gen_msg_filename)
mkdir -p $(dirname $external_msg_filename)
mv $gen_msg_filename $external_msg_filename
ln -s $external_msg_filename $gen_msg_filename

$NOTMUCH new

echo "### Testing \"notmuch new\" with a two-level directory with 3 files"

do_sleep
generate_message two/levels
generate_message two/levels
generate_message two/levels

$NOTMUCH new

echo "### Testing \"notmuch new\" with deletion of two-level directory (3 files)"

do_sleep
rm -rf ${MAIL_DIR}/two

$NOTMUCH new

cat <<EOF
Notmuch test suite complete.

Intermediate state can be examined in:
	${TEST_DIR}
EOF

--=-=-=--