Return-Path: X-Original-To: notmuch@notmuchmail.org Delivered-To: notmuch@notmuchmail.org Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 8015F431E64 for ; Wed, 5 Dec 2012 19:02:13 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0l1H+CRuxSsx for ; Wed, 5 Dec 2012 19:02:11 -0800 (PST) Received: from tesseract.cs.unb.ca (tesseract.cs.unb.ca [131.202.240.238]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 854F2431FB6 for ; Wed, 5 Dec 2012 19:02:11 -0800 (PST) Received: from fctnnbsc30w-142167090129.dhcp-dynamic.fibreop.nb.bellaliant.net ([142.167.90.129] helo=zancas.localnet) by tesseract.cs.unb.ca with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1TgRiy-0002Cz-Pn; Wed, 05 Dec 2012 23:02:09 -0400 Received: from bremner by zancas.localnet with local (Exim 4.80) (envelope-from ) id 1TgRit-0001Yf-A5; Wed, 05 Dec 2012 23:02:03 -0400 From: david@tethera.net To: notmuch@notmuchmail.org Subject: [Patch v2 05/11] perf-test: cache unpacked corpus Date: Wed, 5 Dec 2012 23:01:42 -0400 Message-Id: <1354762908-5788-6-git-send-email-david@tethera.net> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1354762908-5788-1-git-send-email-david@tethera.net> References: <1354762908-5788-1-git-send-email-david@tethera.net> X-Spam_bar: - Cc: David Bremner X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Dec 2012 03:02:13 -0000 From: David Bremner Unpacking is not really the expensive step (compared to the initial notmuch new), but this is a pre-requisite to caching the database. --- performance-test/.gitignore | 1 + performance-test/Makefile.local | 2 +- performance-test/perf-test-lib.sh | 51 ++++++++++++++++++++++++------------- 3 files changed, 36 insertions(+), 18 deletions(-) diff --git a/performance-test/.gitignore b/performance-test/.gitignore index 53f2697..796ed01 100644 --- a/performance-test/.gitignore +++ b/performance-test/.gitignore @@ -1 +1,2 @@ tmp.*/ +corpus/ diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local index 5d2acbd..63e4c3d 100644 --- a/performance-test/Makefile.local +++ b/performance-test/Makefile.local @@ -29,4 +29,4 @@ $(TXZFILE): download-corpus: wget -O ${TXZFILE} ${DEFAULT_URL} -CLEAN := $(CLEAN) $(dir)/tmp.* +CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh index 415bf15..0c10718 100644 --- a/performance-test/perf-test-lib.sh +++ b/performance-test/perf-test-lib.sh @@ -39,33 +39,50 @@ add_email_corpus () { rm -rf ${MAIL_DIR} - case "$1" in - --small) - arg="mail/enron/bailey-s" + case "$corpus_size" in + small) + mail_subdir="mail/enron/bailey-s" + check_for="${TEST_DIRECTORY}/corpus/$mail_subdir" ;; - --medium) - arg="mail/notmuch-archive" + medium) + mail_subdir="mail/notmuch-archive" + check_for="${TEST_DIRECTORY}/corpus/$mail_subdir" ;; *) - arg=mail + mail_subdir=mail + check_for="${TEST_DIRECTORY}/corpus/$mail_subdir/enron/wolfe-j" esac - if command -v pixz > /dev/null; then - XZ=pixz - else - XZ=xz + MAIL_CORPUS="${TEST_DIRECTORY}/corpus/$mail_subdir" + args=() + if [ ! -d "$check_for" ] ; then + args+=("notmuch-email-corpus/$mail_subdir") fi - printf "Unpacking corpus\n" - tar --checkpoint=.5000 --extract --strip-components=1 \ - --directory ${TMP_DIRECTORY} \ - --use-compress-program ${XZ} \ - --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \ - notmuch-email-corpus/"$arg" + if [[ ${#args[@]} > 0 ]]; then + if command -v pixz > /dev/null; then + XZ=pixz + else + XZ=xz + fi + + printf "Unpacking corpus\n" + mkdir -p "${TEST_DIRECTORY}/corpus" + + tar --checkpoint=.5000 --extract --strip-components=1 \ + --directory ${TEST_DIRECTORY}/corpus \ + --use-compress-program ${XZ} \ + --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \ + "${args[@]}" + + printf "\n" - printf "\n" + fi + + cp -lr $MAIL_CORPUS $MAIL_DIR } + print_header () { printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \ ${PERFTEST_VERSION} ${corpus_size} -- 1.7.10.4