From: David Bremner <david@tethera.net>
To: Austin Clements <amdragon@MIT.EDU>
Subject: Re: [Patch v4 2/2] test: initial performance testing infrastructure
In-Reply-To: <20121125214023.GO4562@mit.edu>
References: <1353855745-11697-1-git-send-email-david@tethera.net>
	<1353855745-11697-3-git-send-email-david@tethera.net>
	<20121125214023.GO4562@mit.edu>
User-Agent: Notmuch/0.14+75~g984212d (http://notmuchmail.org) Emacs/24.1.1
	(x86_64-pc-linux-gnu)
Date: Sun, 25 Nov 2012 20:05:42 -0400
Message-ID: <87obilut2h.fsf@zancas.localnet>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam_bar: -
Cc: notmuch@notmuchmail.org
X-BeenThere: notmuch@notmuchmail.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: "Use and development of the notmuch mail system."
	<notmuch.notmuchmail.org>
List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
	<mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
List-Archive: <http://notmuchmail.org/pipermail/notmuch>
List-Post: <mailto:notmuch@notmuchmail.org>
List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
	<mailto:notmuch-request@notmuchmail.org?subject=subscribe>
X-List-Received-Date: Mon, 26 Nov 2012 00:05:53 -0000

Austin Clements <amdragon@MIT.EDU> writes:

>> +subdirs := compat completion emacs lib man parse-time-string
>> +subdirs := $(subdirs) performance-test util test

> += ?
>

Sure.

>> +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz
>
> Would it make sense to split out the different size corpora so a user
> could, say, only download the small one?

Currently the choice of test is local to given test file; one doing
something particularly intense (or just lots of repetitions) might want
to only use a subset. So I'm not sure if separate downloading of smaller
corpora makes sense. This is all hypothetical at the moment, since the
one test file uses the full corpus.

> "\nPlease download ${TXZFILE} using\n\n"?

OK

>> +add_email_corpus takes arguments "--small" and "--medium" for when you
>> +want smaller corpuses to check.
>
> "corpora"?

reworded to say 

,----
| add_email_corpus takes arguments "--small" and "--medium" for when you
| want smaller subsets of the corpus to check.
`----

>
> I'm a bit confused by this.  What happens if you don't specify --small
> or --medium?  Is the "large"/default corpus just the combined small
> and medium corpora?  Would be worth a comment, at least.

Hopefully the README makes this clear(er) now?

> This probably doesn't matter now, but I wonder if we want to unpack on
> first use to somewhere not test-specific and then cp -rl the corpus
> into the test directory.  I haven't tried unpacking the corpus yet,
> but if you're running tests repeatedly to compare results, or running
> more than one performance test, it seems like a full decompress and
> unpack could get onerous.

Hmm. On my machine it is 10s for the copy versus 45s for a full
unpack. For some reason I tested with "cp -a" which is incredibly slow, 
so I thought there was no loss. For comparison the basic test takes
about 10 minutes on the same machine.

In any case this can wait until we have a second test file and a second
call to add_mail_corpus, adding caching now would not help.