Re: [notmuch] Notmuch performance problems on OSX

diff --git a/79/b68e99a8683de7170a8a014d08c7ec99611698 b/79/b68e99a8683de7170a8a014d08c7ec99611698
new file mode 100644 (file)
index 0000000..3268549
--- /dev/null
+++ b/79/b68e99a8683de7170a8a014d08c7ec99611698
@@ -0,0 +1,141 @@
+Return-Path: <cworth@cworth.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id E810C431FBC\r
+       for <notmuch@notmuchmail.org>; Thu, 14 Jan 2010 15:16:26 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.198\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.198 tagged_above=-999 required=5\r
+       tests=[ALL_TRUSTED=-1.8, AWL=0.342, BAYES_20=-0.74] autolearn=ham\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id zilSrVTyYztG; Thu, 14 Jan 2010 15:16:25 -0800 (PST)\r
+Received: from yoom.home.cworth.org (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id C5D62431FAE;\r
+       Thu, 14 Jan 2010 15:16:25 -0800 (PST)\r
+Received: by yoom.home.cworth.org (Postfix, from userid 1000)\r
+       id 2241C550090; Thu, 14 Jan 2010 15:16:25 -0800 (PST)\r
+From: Carl Worth <cworth@cworth.org>\r
+To: Oliver Charles <oliver.g.charles@googlemail.com>, notmuch@notmuchmail.org\r
+In-Reply-To: <5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com>\r
+References: <5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com>\r
+Date: Thu, 14 Jan 2010 15:16:24 -0800\r
+Message-ID: <87d41ce0qf.fsf@yoom.home.cworth.org>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; boundary="=-=-=";\r
+       micalg=pgp-sha1; protocol="application/pgp-signature"\r
+Subject: Re: [notmuch] Notmuch performance problems on OSX\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 14 Jan 2010 23:16:27 -0000\r
+\r
+--=-=-=\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+Hi Oliver, welcome to notmuch!\r
+\r
+On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles <oliver.g.charles@google=\r
+mail.com> wrote:\r
+> I've installed the latest notmuch from Git at this time of writing,\r
+> along with Xapian from SVN head. However, just tagging a single thread\r
+> with only one message seems to take too long:\r
+>=20\r
+> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221\r
+>=20\r
+> real 0m0.812s\r
+> user 0m0.022s\r
+> sys  0m0.037s\r
+\r
+Things work quite a bit faster than that on my machine:\r
+\r
+$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mai=\r
+l.gmail.com\r
+\r
+real   0m0.024s\r
+user   0m0.012s\r
+sys    0m0.004s\r
+\r
+But that could just be system differences.\r
+\r
+> And tagging all my messages is really horrible:\r
+>=20\r
+> $ time notmuch tag +foobar tag:inbox\r
+>=20\r
+> real 0m5.076s\r
+> user 0m3.688s\r
+> sys  0m0.105s\r
+\r
+For this operation, I can't really compare. How many messages are you\r
+tagging? Here's that operation for me with 525 messages in my inbox:\r
+\r
+$ time notmuch tag +foobar tag:inbox\r
+\r
+real   0m1.551s\r
+user   0m1.504s\r
+sys    0m0.016s\r
+\r
+> That xapian-svn was built from svn HEAD right now, so I'm assuming it\r
+> contains the #250 fix (http://trac.xapian.org/changeset/13808)\r
+\r
+Which I think means that things could have been even *much* slower\r
+before. ;-)\r
+\r
+The Xapian defect #250 was just one, initial (and obvious) performance\r
+problem. [Though, as I mentioned in a previous thread, if you're using a\r
+Xapian flint database, (look for .notmuch/xapian/iamflint), then you\r
+won't get the benefit of the Xapian fix until you rebuild your notmuch\r
+database from scratch with a current notmuch.]\r
+\r
+Once you've verified that you've got the #250 fix functional, there\r
+could still be lots of performance bugs. And it would be time to start\r
+profiling.\r
+\r
+Perhaps the "notmuch daemon" idea (which we've proposed earlier for\r
+other reasons) could help reduce overhead from reading the database and\r
+writing it back out again. So that might be one avenue to explore for\r
+fixing things.\r
+\r
+I have no idea what OS X does, but Linux keeps my notmuch database in\r
+its buffer cache so I can do these operations without even touching\r
+"disk" (which is actually an SSD anyway, which also helps). I just\r
+tried, and was able to get the single-message tag operation to be 3\r
+times slower by dropping the cache:\r
+\r
+$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches "\r
+$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mai=\r
+l.gmail.com\r
+\r
+real   0m0.062s\r
+user   0m0.000s\r
+sys    0m0.020s\r
+\r
+But again, whatever the performance problem might be, the first step\r
+would be to examine some profiles. (And I'm clueless, myself as to what\r
+profiling tools might be available for OS X.)\r
+\r
+=2DCarl\r
+\r
+--=-=-=\r
+Content-Type: application/pgp-signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.10 (GNU/Linux)\r
+\r
+iD8DBQFLT6XI6JDdNq8qSWgRAvqAAJ9YBzxsYXYrJ6qcg52b+Loik5Up+ACfbgjd\r
+G7XrYeT28dru4gDwi5DP6eU=\r
+=Rico\r
+-----END PGP SIGNATURE-----\r
+--=-=-=--\r