Re: [PATCH 3/4] Optimize thread search using matched docid sets.

diff --git a/f7/d68306f3c9f0476c992e4da6471c3caec83a88 b/f7/d68306f3c9f0476c992e4da6471c3caec83a88
new file mode 100644 (file)
index 0000000..e465f4a
--- /dev/null
+++ b/f7/d68306f3c9f0476c992e4da6471c3caec83a88
@@ -0,0 +1,122 @@
+Return-Path: <cworth@cworth.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id D42D2429E24\r
+       for <notmuch@notmuchmail.org>; Fri, 28 Jan 2011 14:02:54 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.99\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.99 tagged_above=-999 required=5\r
+       tests=[ALL_TRUSTED=-1, T_MIME_NO_TEXT=0.01] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id Fy4UErepAb2g; Fri, 28 Jan 2011 14:02:53 -0800 (PST)\r
+Received: from yoom.home.cworth.org (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id ECD9C429E21;\r
+       Fri, 28 Jan 2011 14:02:50 -0800 (PST)\r
+Received: by yoom.home.cworth.org (Postfix, from userid 1000)\r
+       id 7B63B25449C; Sat, 29 Jan 2011 07:26:44 +1000 (EST)\r
+From: Carl Worth <cworth@cworth.org>\r
+To: Austin Clements <amdragon@MIT.EDU>\r
+Subject: Re: [PATCH 3/4] Optimize thread search using matched docid sets.\r
+In-Reply-To: <20101208215844.GS2447@mit.edu>\r
+References: <20101117192826.GU2439@mit.edu>\r
+       <874oap5aek.fsf@yoom.home.cworth.org>\r
+       <20101208215844.GS2447@mit.edu>\r
+User-Agent: Notmuch/0.5 (http://notmuchmail.org) Emacs/23.2.1\r
+       (i486-pc-linux-gnu)\r
+Date: Sat, 29 Jan 2011 07:26:44 +1000\r
+Message-ID: <878vy4n2i3.fsf@yoom.home.cworth.org>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/mixed; boundary="=-=-="\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 28 Jan 2011 22:02:55 -0000\r
+\r
+--=-=-=\r
+Content-Type: multipart/signed; boundary="==-=-=";\r
+       micalg=pgp-sha1; protocol="application/pgp-signature"\r
+\r
+--==-=-=\r
+\r
+On Wed, 8 Dec 2010 16:58:44 -0500, Austin Clements <amdragon@MIT.EDU> wrote:\r
+> Now that this is in (and I have a temporary respite from TA duties),\r
+> I'm going to finish up and send out my other ~1.7X improvement, just\r
+> to get it out of my queue.  Then I'll look at making a performance\r
+> regression suite.  Were you thinking of some standard set of timed\r
+> operations wrapped in a little script that can tell you if you've made\r
+> things worse, or something more elaborate?\r
+\r
+I recently started making a perf/notmuch-perf script for notmuch (see\r
+below). I was doing this in preparation for my linux.conf.au talk on\r
+notmuch, (though I ended up not talking about performance in concrete\r
+terms).\r
+\r
+I don't know how much further I'll run with this now, but if this is a\r
+useful starting place for anyone, let me know and I can obviously add\r
+this to the repository.\r
+\r
+So the idea with this script is that the timed operations actually\r
+depend on local data, (your current mail collection as indicated by\r
+NOTMUCH_CONFIG). So the operations aren't standardized to enable\r
+comparison between different people, (unless they also agree on some\r
+common mail collection).\r
+\r
+My script as attached runs only "notmuch new" to time the original\r
+indexing. Beyond that I'd like to time some common operations,\r
+(adding a new message, searching for a single message, searching for\r
+many messages, searching for all messages, etc.).\r
+\r
+And then on top of this, I'd like to have a little utility that could\r
+compare several different runs captured previously. That would let me do\r
+the regression testing I'd like to ensure we never make performance\r
+worse.\r
+\r
+Please feel free to run with this or with your own approach as you see\r
+fit.\r
+\r
+-Carl\r
+\r
+\r
+--==-=-=\r
+Content-Type: application/pgp-signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.10 (GNU/Linux)\r
+\r
+iD8DBQFNQzSU6JDdNq8qSWgRArrAAJ4x4Pe7Va/tR5F3K4zqS5SiWVldDwCfcASp\r
+Ylw7s9u/yVpNovNFPqM/GZ0=\r
+=qG06\r
+-----END PGP SIGNATURE-----\r
+--==-=-=--\r
+\r
+--=-=-=\r
+Content-Type: application/octet-stream; charset=utf-8\r
+Content-Disposition: attachment; filename=notmuch-perf\r
+Content-Transfer-Encoding: base64\r
+\r
+IyEvYmluL3NoCnNldCAtZQoKU0NSQVRDSD0iJChwd2QpL25vdG11Y2gtcGVyZl8kKGRhdGUgKyVG\r
+XyVUKSIKQ09ORklHX09SSUc9JHtOT1RNVUNIX0NPTkZJRzotJEhPTUUvLm5vdG11Y2gtY29uZmln\r
+fQoKaWYgISBbIC1mICIkQ09ORklHX09SSUciIF07IHRoZW4KICAgZWNobyAiRXJyb3I6IE5vIGNv\r
+bmZpZ3VyYXRpb24gZm91bmQgYXQgJENPTkZJR19PUklHIgogICBlY2hvICJQbGVhc2UgcHJvdmlk\r
+ZSBhIGRlZmF1bHQgY29uZmlndXJhdGlvbiBmaWxlIHRvIHNwZWNpZnkgdGhlIG1haWwgdG8gaW5k\r
+ZXguIgogICBleGl0IDEKZmkKCm1rZGlyICIkU0NSQVRDSCIKY2QgIiRTQ1JBVENIIgoKTUFJTF9Q\r
+QVRIX09SSUc9JChub3RtdWNoIGNvbmZpZyBnZXQgZGF0YWJhc2UucGF0aCkKTUFJTF9QQVRIPSIk\r
+U0NSQVRDSC9tYWlsIgoKZXhwb3J0IE5PVE1VQ0hfQ09ORklHPSIkU0NSQVRDSC9ub3RtdWNoLWNv\r
+bmZpZyIKY3AgIiRDT05GSUdfT1JJRyIgIiROT1RNVUNIX0NPTkZJRyIKCmxuIC1zICIkTUFJTF9Q\r
+QVRIX09SSUciICIkTUFJTF9QQVRIIgoKZWNobyAiTm93IHJ1bm5pbmcgbm90bXVjaCBuZXcgdG8g\r
+aW5kZXggYWxsIG1haWwiCnRpbWUgbm90bXVjaCBuZXcKCg==\r
+--=-=-=--\r