From: David Bremner Date: Mon, 26 Nov 2012 00:05:42 +0000 (+2000) Subject: Re: [Patch v4 2/2] test: initial performance testing infrastructure X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=917b593b7e59bbb366344bfd5b57f644486405bb;p=notmuch-archives.git Re: [Patch v4 2/2] test: initial performance testing infrastructure --- diff --git a/9e/2b6fdd5ab0618967b42ae38728bd8ed1f444fc b/9e/2b6fdd5ab0618967b42ae38728bd8ed1f444fc new file mode 100644 index 000000000..d0ad22ed7 --- /dev/null +++ b/9e/2b6fdd5ab0618967b42ae38728bd8ed1f444fc @@ -0,0 +1,117 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 08D53431FAF + for ; Sun, 25 Nov 2012 16:05:53 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id DG3h5vQDF2tl for ; + Sun, 25 Nov 2012 16:05:52 -0800 (PST) +Received: from tesseract.cs.unb.ca (tesseract.cs.unb.ca [131.202.240.238]) + (using TLSv1 with cipher AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 78C52431FAE + for ; Sun, 25 Nov 2012 16:05:52 -0800 (PST) +Received: from fctnnbsc30w-156034089108.dhcp-dynamic.fibreop.nb.bellaliant.net + ([156.34.89.108] helo=zancas.localnet) + by tesseract.cs.unb.ca with esmtpsa + (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) + (envelope-from ) + id 1TcmCs-0007pZ-Vu; Sun, 25 Nov 2012 20:05:51 -0400 +Received: from bremner by zancas.localnet with local (Exim 4.80) + (envelope-from ) + id 1TcmCm-0005SZ-JH; Sun, 25 Nov 2012 20:05:44 -0400 +From: David Bremner +To: Austin Clements +Subject: Re: [Patch v4 2/2] test: initial performance testing infrastructure +In-Reply-To: <20121125214023.GO4562@mit.edu> +References: <1353855745-11697-1-git-send-email-david@tethera.net> + <1353855745-11697-3-git-send-email-david@tethera.net> + <20121125214023.GO4562@mit.edu> +User-Agent: Notmuch/0.14+75~g984212d (http://notmuchmail.org) Emacs/24.1.1 + (x86_64-pc-linux-gnu) +Date: Sun, 25 Nov 2012 20:05:42 -0400 +Message-ID: <87obilut2h.fsf@zancas.localnet> +MIME-Version: 1.0 +Content-Type: text/plain +X-Spam_bar: - +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 26 Nov 2012 00:05:53 -0000 + +Austin Clements writes: + +>> +subdirs := compat completion emacs lib man parse-time-string +>> +subdirs := $(subdirs) performance-test util test + +> += ? +> + +Sure. + +>> +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz +> +> Would it make sense to split out the different size corpora so a user +> could, say, only download the small one? + +Currently the choice of test is local to given test file; one doing +something particularly intense (or just lots of repetitions) might want +to only use a subset. So I'm not sure if separate downloading of smaller +corpora makes sense. This is all hypothetical at the moment, since the +one test file uses the full corpus. + +> "\nPlease download ${TXZFILE} using\n\n"? + +OK + +>> +add_email_corpus takes arguments "--small" and "--medium" for when you +>> +want smaller corpuses to check. +> +> "corpora"? + +reworded to say + +,---- +| add_email_corpus takes arguments "--small" and "--medium" for when you +| want smaller subsets of the corpus to check. +`---- + +> +> I'm a bit confused by this. What happens if you don't specify --small +> or --medium? Is the "large"/default corpus just the combined small +> and medium corpora? Would be worth a comment, at least. + +Hopefully the README makes this clear(er) now? + +> This probably doesn't matter now, but I wonder if we want to unpack on +> first use to somewhere not test-specific and then cp -rl the corpus +> into the test directory. I haven't tried unpacking the corpus yet, +> but if you're running tests repeatedly to compare results, or running +> more than one performance test, it seems like a full decompress and +> unpack could get onerous. + +Hmm. On my machine it is 10s for the copy versus 45s for a full +unpack. For some reason I tested with "cp -a" which is incredibly slow, +so I thought there was no loss. For comparison the basic test takes +about 10 minutes on the same machine. + +In any case this can wait until we have a second test file and a second +call to add_mail_corpus, adding caching now would not help.