--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 10A75431FAF\r
+ for <notmuch@notmuchmail.org>; Sun, 25 Nov 2012 13:40:29 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id hwi1UAfsSmTO for <notmuch@notmuchmail.org>;\r
+ Sun, 25 Nov 2012 13:40:28 -0800 (PST)\r
+Received: from dmz-mailsec-scanner-2.mit.edu (DMZ-MAILSEC-SCANNER-2.MIT.EDU\r
+ [18.9.25.13])\r
+ by olra.theworths.org (Postfix) with ESMTP id D1AEB431FAE\r
+ for <notmuch@notmuchmail.org>; Sun, 25 Nov 2012 13:40:27 -0800 (PST)\r
+X-AuditID: 1209190d-b7f906d0000008de-d4-50b2904b98d3\r
+Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])\r
+ by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 88.19.02270.B4092B05; Sun, 25 Nov 2012 16:40:27 -0500 (EST)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id qAPLeQR9008328; \r
+ Sun, 25 Nov 2012 16:40:26 -0500\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id qAPLeOjh003344\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+ Sun, 25 Nov 2012 16:40:25 -0500 (EST)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1Tcjw8-0004MD-13; Sun, 25 Nov 2012 16:40:24 -0500\r
+Date: Sun, 25 Nov 2012 16:40:24 -0500\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: david@tethera.net\r
+Subject: Re: [Patch v4 2/2] test: initial performance testing infrastructure\r
+Message-ID: <20121125214023.GO4562@mit.edu>\r
+References: <1353855745-11697-1-git-send-email-david@tethera.net>\r
+ <1353855745-11697-3-git-send-email-david@tethera.net>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <1353855745-11697-3-git-send-email-david@tethera.net>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFprBKsWRmVeSWpSXmKPExsUixG6nrus9YVOAwb1WUYuNy34yW9xo7Wa0\r
+ uH5zJrMDs8evtrnMHs9W3WL22HLoPXMAcxSXTUpqTmZZapG+XQJXxsKDx5gKOkMqVnd3MzUw\r
+ vnXsYuTkkBAwkZj7djsThC0mceHeerYuRi4OIYF9jBJ9a5exgySEBDYwSrw+xwuRuMgkce/Q\r
+ TDaIxBJGiVkvM0BsFgFVicaFB1hBbDYBDYlt+5czgtgiAqISpzdNAoszC9hLLJo9CWyosICP\r
+ xLlds8A28wpoS0y98R9qWaXE5/mvmSHighInZz5hgejVkrjx7yVQPQeQLS2x/B8HSJhTwEli\r
+ zpoNYCWiAioSU05uY5vAKDQLSfcsJN2zELoXMDKvYpRNya3SzU3MzClOTdYtTk7My0st0jXS\r
+ y80s0UtNKd3ECAp0TkneHYzvDiodYhTgYFTi4b2RuDFAiDWxrLgy9xCjJAeTkihvc+umACG+\r
+ pPyUyozE4oz4otKc1OJDjBIczEoivBOZgHK8KYmVValF+TApaQ4WJXHeKyk3/YUE0hNLUrNT\r
+ UwtSi2CyMhwcShK8Bv1AjYJFqempFWmZOSUIaSYOTpDhPEDDU0BqeIsLEnOLM9Mh8qcYdTl6\r
+ /3U9YRRiycvPS5US57UFKRIAKcoozYObA0tQrxjFgd4S5k0GqeIBJje4Sa+AljABLUm+vhFk\r
+ SUkiQkqqgbHzq6PKyh2L695Gad5t7bvtzfK93InNsS7av7Ct5eb+j3t94ufzFlW9vxb1R2aL\r
+ v48d0/EHf1J6F975c13U1El6zr+yy4Y2xWwegZ8XmVauLGB5t+Tkkdtzs9xkZLcvcplQt2v5\r
+ moz2L43/2nZ/E/15xEpt93KxEwrPl7xmnZp/Qyq674F24mwlluKMREMt5qLiRACFNW0uKwMA AA==\r
+Cc: notmuch@notmuchmail.org, David Bremner <bremner@debian.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 25 Nov 2012 21:40:29 -0000\r
+\r
+Quoth david@tethera.net on Nov 25 at 11:02 am:\r
+> From: David Bremner <bremner@debian.org>\r
+> \r
+> This is not near as fancy as as the unit tests, on the theory that\r
+> the code should typically be crashing when performance tuning.\r
+> Nonetheless, there is plenty of room for improvement. Several more of\r
+> the pieces of the test infrastructure (e.g. the option parsing) could\r
+> be factored out into test/test-lib-common.sh\r
+> ---\r
+> Makefile | 3 +-\r
+> performance-test/.gitignore | 1 +\r
+> performance-test/Makefile | 7 +++\r
+> performance-test/Makefile.local | 32 ++++++++++\r
+> performance-test/README | 50 +++++++++++++++\r
+> performance-test/basic | 15 +++++\r
+> performance-test/download/.gitignore | 2 +\r
+> .../download/notmuch-email-corpus-0.2.tar.xz.asc | 9 +++\r
+> performance-test/notmuch-perf-test | 25 ++++++++\r
+> performance-test/perf-test-lib.sh | 65 ++++++++++++++++++++\r
+> performance-test/version.sh | 3 +\r
+> 11 files changed, 211 insertions(+), 1 deletion(-)\r
+> create mode 100644 performance-test/.gitignore\r
+> create mode 100644 performance-test/Makefile\r
+> create mode 100644 performance-test/Makefile.local\r
+> create mode 100644 performance-test/README\r
+> create mode 100755 performance-test/basic\r
+> create mode 100644 performance-test/download/.gitignore\r
+> create mode 100644 performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc\r
+> create mode 100755 performance-test/notmuch-perf-test\r
+> create mode 100644 performance-test/perf-test-lib.sh\r
+> create mode 100644 performance-test/version.sh\r
+> \r
+> diff --git a/Makefile b/Makefile\r
+> index bb9c316..5decbea 100644\r
+> --- a/Makefile\r
+> +++ b/Makefile\r
+> @@ -3,7 +3,8 @@\r
+> all:\r
+> \r
+> # List all subdirectories here. Each contains its own Makefile.local\r
+> -subdirs = compat completion emacs lib man parse-time-string util test\r
+> +subdirs := compat completion emacs lib man parse-time-string\r
+> +subdirs := $(subdirs) performance-test util test\r
+\r
++= ?\r
+\r
+> \r
+> # We make all targets depend on the Makefiles themselves.\r
+> global_deps = Makefile Makefile.config Makefile.local \\r
+> diff --git a/performance-test/.gitignore b/performance-test/.gitignore\r
+> new file mode 100644\r
+> index 0000000..53f2697\r
+> --- /dev/null\r
+> +++ b/performance-test/.gitignore\r
+> @@ -0,0 +1 @@\r
+> +tmp.*/\r
+> diff --git a/performance-test/Makefile b/performance-test/Makefile\r
+> new file mode 100644\r
+> index 0000000..de492a7\r
+> --- /dev/null\r
+> +++ b/performance-test/Makefile\r
+> @@ -0,0 +1,7 @@\r
+> +# See Makefile.local for the list of files to be compiled in this\r
+> +# directory.\r
+> +all:\r
+> + $(MAKE) -C .. all\r
+> +\r
+> +.DEFAULT:\r
+> + $(MAKE) -C .. $@\r
+> diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local\r
+> new file mode 100644\r
+> index 0000000..1114ec1\r
+> --- /dev/null\r
+> +++ b/performance-test/Makefile.local\r
+> @@ -0,0 +1,32 @@\r
+> +# -*- makefile -*-\r
+> +\r
+> +dir := performance-test\r
+> +\r
+> +include $(dir)/version.sh\r
+> +\r
+> +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz\r
+\r
+Would it make sense to split out the different size corpora so a user\r
+could, say, only download the small one?\r
+\r
+> +TXZFILE := ${dir}/download/${CORPUS_NAME}\r
+> +SIGFILE := ${TXZFILE}.asc\r
+> +TEST_SCRIPT := ${dir}/notmuch-perf-test\r
+> +DEFAULT_URL := http://notmuchmail.org/releases/${CORPUS_NAME}\r
+> +\r
+> +perf-test: setup-perf-test all\r
+> + $(TEST_SCRIPT) $(OPTIONS)\r
+> +\r
+> +.PHONY: download-corpus setup-perf-test\r
+> +\r
+> +# Note that this intentionally does not depend on download-corpus.\r
+> +setup-perf-test: $(TXZFILE)\r
+> + gpg --verify $(SIGFILE)\r
+> +\r
+> +$(TXZFILE):\r
+> + @printf "\nPlease download ${TXZFILE}.\n\n"\r
+\r
+"\nPlease download ${TXZFILE} using\n\n"?\r
+\r
+> + @printf "\t%% make download-corpus\n\n"\r
+> + @echo or see http://notmuchmail.org/corpus for download locations\r
+> + @echo\r
+> + @false\r
+> +\r
+> +download-corpus:\r
+> + wget -O ${TXZFILE} ${DEFAULT_URL}\r
+> +\r
+> +CLEAN := $(CLEAN) $(dir)/tmp.*\r
+> diff --git a/performance-test/README b/performance-test/README\r
+> new file mode 100644\r
+> index 0000000..239d2fb\r
+> --- /dev/null\r
+> +++ b/performance-test/README\r
+> @@ -0,0 +1,50 @@\r
+> +Pre-requisites\r
+> +--------------\r
+> +\r
+> +In addition to having notmuch, you need:\r
+> +\r
+> +- gpg\r
+> +- gnu tar\r
+> +- gnu time\r
+> +- xz. Some speedup can be gotten by installing "pixz", but this is\r
+> + probably only worthwhile if you are debugging the tests.\r
+> +\r
+> +Getting set up to run tests:\r
+> +----------------------------\r
+> +\r
+> +First, you need to get the corpus.\r
+> +\r
+> +It should work to run\r
+> +\r
+> + % make download-corpus\r
+> +\r
+> +In case that fails or is too slow, check\r
+> +\r
+> + http://notmuchmail.org/corpus\r
+> +\r
+> +for a list of mirrors.\r
+> +\r
+> +Running tests\r
+> +-------------\r
+> +\r
+> +The easiest way to run performance tests is to say "make perf-test", (or\r
+> +simply run the notmuch-perf-test script). Either command will run all\r
+> +available performance tests.\r
+> +\r
+> +Alternately, you can run a specific subset of tests by simply invoking\r
+> +one of the executable scripts in this directory, (such as ./basic).\r
+> +\r
+> +\r
+> +Writing tests\r
+> +-------------\r
+> +\r
+> +Have a look at "basic" for an example.\r
+> +\r
+> +add_email_corpus takes arguments "--small" and "--medium" for when you\r
+> +want smaller corpuses to check.\r
+\r
+"corpora"?\r
+\r
+> +\r
+> +time_done does the cleanup; comment it out or define "$debug" to leave\r
+> +the temporary files around.\r
+> +\r
+> +Currently there is no option processing (e.g. --debug) in the\r
+> +performance tests.\r
+> diff --git a/performance-test/basic b/performance-test/basic\r
+> new file mode 100755\r
+> index 0000000..9d015ee\r
+> --- /dev/null\r
+> +++ b/performance-test/basic\r
+> @@ -0,0 +1,15 @@\r
+> +#!/bin/bash\r
+> +\r
+> +. ./perf-test-lib.sh\r
+> +\r
+> +add_email_corpus\r
+> +\r
+> +print_header\r
+> +\r
+> +time_run 'initial notmuch new' 'notmuch new'\r
+> +time_run 'second notmuch new' 'notmuch new'\r
+> +time_run 'dump *' 'notmuch dump > tags.out'\r
+> +time_run 'restore *' 'notmuch restore < tags.out'\r
+> +time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"\r
+> +\r
+> +time_done\r
+> diff --git a/performance-test/download/.gitignore b/performance-test/download/.gitignore\r
+> new file mode 100644\r
+> index 0000000..7b09234\r
+> --- /dev/null\r
+> +++ b/performance-test/download/.gitignore\r
+> @@ -0,0 +1,2 @@\r
+> +*.tar.gz\r
+> +*.tar.xz\r
+> diff --git a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc\r
+> new file mode 100644\r
+> index 0000000..c8b4b3d\r
+> --- /dev/null\r
+> +++ b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc\r
+> @@ -0,0 +1,9 @@\r
+> +-----BEGIN PGP SIGNATURE-----\r
+> +Version: GnuPG v1.4.12 (GNU/Linux)\r
+> +\r
+> +iJwEAAECAAYFAlCsvx0ACgkQTiiN/0Um85kZAwP9GgOQ22jK8mr5X4pT/mB8EjSH\r
+> +QbndlxxbRrP0ChTqjBQoD3IsTHjNL7W572BfXb/MNo94R/iIQ7yTHCDVNuwBhvKd\r
+> +7qgIuW2FUS1uTfJRP5KBNf8JPuin+6wqGe8/+y/iOs+XJSdiYg1ElS49Ntnpg0yl\r
+> +btImgEcxTxQ2qfzDS1g=\r
+> +=iuZR\r
+> +-----END PGP SIGNATURE-----\r
+> diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test\r
+> new file mode 100755\r
+> index 0000000..1bea345\r
+> --- /dev/null\r
+> +++ b/performance-test/notmuch-perf-test\r
+> @@ -0,0 +1,25 @@\r
+> +#!/usr/bin/env bash\r
+> +\r
+> +# Run tests\r
+> +#\r
+> +# Copyright (c) 2005 Junio C Hamano\r
+> +#\r
+> +# Adapted from a Makefile to a shell script by Carl Worth (2010)\r
+> +\r
+> +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then\r
+> + echo "Error: The notmuch test suite requires a bash version >= 4.0"\r
+> + echo "due to use of associative arrays within the test suite."\r
+> + echo "Please try again with a newer bash (or help us fix the"\r
+> + echo "test suite to be more portable). Thanks."\r
+> + exit 1\r
+> +fi\r
+> +\r
+> +cd $(dirname "$0")\r
+> +\r
+> +TESTS="\r
+> + basic\r
+> +"\r
+> +\r
+> +for test in $TESTS; do\r
+> + ./$test "$@"\r
+> +done\r
+> diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh\r
+> new file mode 100644\r
+> index 0000000..11ccc77\r
+> --- /dev/null\r
+> +++ b/performance-test/perf-test-lib.sh\r
+> @@ -0,0 +1,65 @@\r
+> +. ./version.sh\r
+> +\r
+> +. ../test/test-lib-common.sh\r
+> +\r
+> +set -e\r
+> +\r
+> +if ! test -x ../notmuch\r
+> +then\r
+> + echo >&2 'You do not seem to have built notmuch yet.'\r
+> + exit 1\r
+> +fi\r
+> +\r
+> +add_email_corpus ()\r
+> +{\r
+> + rm -rf ${MAIL_DIR}\r
+> +\r
+> + arg=""\r
+> + case "$1" in\r
+> + --small)\r
+> + arg="/enron/bailey-s"\r
+> + ;;\r
+> + --medium)\r
+> + arg="/notmuch-archive"\r
+> + ;;\r
+> + esac\r
+> +\r
+> + if command -v pixz > /dev/null; then\r
+> + XZ=pixz\r
+> + else\r
+> + XZ=xz\r
+> + fi\r
+> +\r
+> + printf "Unpacking corpus\n"\r
+> + tar --checkpoint=.5000 --extract --strip-components=1 \\r
+> + --directory ${TMP_DIRECTORY} \\r
+> + --use-compress-program ${XZ} \\r
+> + --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \\r
+> + notmuch-email-corpus/mail"$arg"\r
+\r
+I'm a bit confused by this. What happens if you don't specify --small\r
+or --medium? Is the "large"/default corpus just the combined small\r
+and medium corpora? Would be worth a comment, at least.\r
+\r
+This probably doesn't matter now, but I wonder if we want to unpack on\r
+first use to somewhere not test-specific and then cp -rl the corpus\r
+into the test directory. I haven't tried unpacking the corpus yet,\r
+but if you're running tests repeatedly to compare results, or running\r
+more than one performance test, it seems like a full decompress and\r
+unpack could get onerous.\r
+\r
+> +\r
+> + printf "\n"\r
+> +}\r
+> +\r
+> +print_header () {\r
+> + printf " Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn\tOut\n"\r
+\r
+Should the header include which corpus size is in use? This would be\r
+important when emailing around performance results.\r
+\r
+> +}\r
+> +\r
+> +time_run () {\r
+> + printf "%-22s" "$1"\r
+> + if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi\r
+> + if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then\r
+> + test_failure=$(($test_failure + 1))\r
+> + fi\r
+> +}\r
+> +\r
+> +time_done () {\r
+> + if [ "$test_failure" = "0" ]; then\r
+> + rm -rf "$remove_tmp"\r
+> + exit 0\r
+> + else\r
+> + exit 1\r
+> + fi\r
+> +}\r
+> +\r
+> +cd -P "$test" || error "Cannot setup test environment"\r
+> +test_failure=0\r
+> diff --git a/performance-test/version.sh b/performance-test/version.sh\r
+> new file mode 100644\r
+> index 0000000..d9270b1\r
+> --- /dev/null\r
+> +++ b/performance-test/version.sh\r
+> @@ -0,0 +1,3 @@\r
+> +# this should be both a valid Makefile fragment and valid POSIX(ish) shell.\r
+> +\r
+> +PERFTEST_VERSION=0.2\r
projects / notmuch-archives.git / commitdiff