From: Carl Worth <cworth@cworth.org>
To: Austin Clements <amdragon@MIT.EDU>
Subject: Re: [PATCH 3/4] Optimize thread search using matched docid sets.
In-Reply-To: <20101208215844.GS2447@mit.edu>
References: <20101117192826.GU2439@mit.edu>
	<874oap5aek.fsf@yoom.home.cworth.org>
	<20101208215844.GS2447@mit.edu>
User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.2.1
	(i486-pc-linux-gnu)
Date: Sat, 29 Jan 2011 07:26:44 +1000
Message-ID: <878vy4n2i3.fsf@yoom.home.cworth.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Cc: notmuch@notmuchmail.org
Precedence: list

--=-=-=
Content-Type: multipart/signed; boundary="==-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"

--==-=-=

On Wed, 8 Dec 2010 16:58:44 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Now that this is in (and I have a temporary respite from TA duties),
> I'm going to finish up and send out my other ~1.7X improvement, just
> to get it out of my queue.  Then I'll look at making a performance
> regression suite.  Were you thinking of some standard set of timed
> operations wrapped in a little script that can tell you if you've made
> things worse, or something more elaborate?

I recently started making a perf/notmuch-perf script for notmuch (see
below). I was doing this in preparation for my linux.conf.au talk on
notmuch, (though I ended up not talking about performance in concrete
terms).

I don't know how much further I'll run with this now, but if this is a
useful starting place for anyone, let me know and I can obviously add
this to the repository.

So the idea with this script is that the timed operations actually
depend on local data, (your current mail collection as indicated by
NOTMUCH_CONFIG). So the operations aren't standardized to enable
comparison between different people, (unless they also agree on some
common mail collection).

My script as attached runs only "notmuch new" to time the original
indexing. Beyond that I'd like to time some common operations,
(adding a new message, searching for a single message, searching for
many messages, searching for all messages, etc.).

And then on top of this, I'd like to have a little utility that could
compare several different runs captured previously. That would let me do
the regression testing I'd like to ensure we never make performance
worse.

Please feel free to run with this or with your own approach as you see
fit.

-Carl


--==-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iD8DBQFNQzSU6JDdNq8qSWgRArrAAJ4x4Pe7Va/tR5F3K4zqS5SiWVldDwCfcASp
Ylw7s9u/yVpNovNFPqM/GZ0=
=qG06
-----END PGP SIGNATURE-----
--==-=-=--

--=-=-=
Content-Type: application/octet-stream; charset=utf-8
Content-Disposition: attachment; filename=notmuch-perf
Content-Transfer-Encoding: base64

IyEvYmluL3NoCnNldCAtZQoKU0NSQVRDSD0iJChwd2QpL25vdG11Y2gtcGVyZl8kKGRhdGUgKyVG
XyVUKSIKQ09ORklHX09SSUc9JHtOT1RNVUNIX0NPTkZJRzotJEhPTUUvLm5vdG11Y2gtY29uZmln
fQoKaWYgISBbIC1mICIkQ09ORklHX09SSUciIF07IHRoZW4KICAgZWNobyAiRXJyb3I6IE5vIGNv
bmZpZ3VyYXRpb24gZm91bmQgYXQgJENPTkZJR19PUklHIgogICBlY2hvICJQbGVhc2UgcHJvdmlk
ZSBhIGRlZmF1bHQgY29uZmlndXJhdGlvbiBmaWxlIHRvIHNwZWNpZnkgdGhlIG1haWwgdG8gaW5k
ZXguIgogICBleGl0IDEKZmkKCm1rZGlyICIkU0NSQVRDSCIKY2QgIiRTQ1JBVENIIgoKTUFJTF9Q
QVRIX09SSUc9JChub3RtdWNoIGNvbmZpZyBnZXQgZGF0YWJhc2UucGF0aCkKTUFJTF9QQVRIPSIk
U0NSQVRDSC9tYWlsIgoKZXhwb3J0IE5PVE1VQ0hfQ09ORklHPSIkU0NSQVRDSC9ub3RtdWNoLWNv
bmZpZyIKY3AgIiRDT05GSUdfT1JJRyIgIiROT1RNVUNIX0NPTkZJRyIKCmxuIC1zICIkTUFJTF9Q
QVRIX09SSUciICIkTUFJTF9QQVRIIgoKZWNobyAiTm93IHJ1bm5pbmcgbm90bXVjaCBuZXcgdG8g
aW5kZXggYWxsIG1haWwiCnRpbWUgbm90bXVjaCBuZXcKCg==
--=-=-=--