slowdown in notmuch perf suite with xapian 1.3.5

diff --git a/90/289c305231fc0720668f8aa5fb2ab4851cd985 b/90/289c305231fc0720668f8aa5fb2ab4851cd985
new file mode 100644 (file)
index 0000000..62bc5f3
--- /dev/null
+++ b/90/289c305231fc0720668f8aa5fb2ab4851cd985
@@ -0,0 +1,114 @@
+Return-Path: <david@tethera.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 1D3B66DE0130\r
+ for <notmuch@notmuchmail.org>; Thu,  7 Apr 2016 04:57:00 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.02\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.02 tagged_above=-999 required=5 tests=[AWL=-0.009,\r
+  SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id CHXTUKvdyGcA for <notmuch@notmuchmail.org>;\r
+ Thu,  7 Apr 2016 04:56:51 -0700 (PDT)\r
+Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197])\r
+ by arlo.cworth.org (Postfix) with ESMTPS id BDCA16DE009B\r
+ for <notmuch@notmuchmail.org>; Thu,  7 Apr 2016 04:56:51 -0700 (PDT)\r
+Received: from remotemail by fethera.tethera.net with local (Exim 4.84)\r
+ (envelope-from <david@tethera.net>)\r
+ id 1ao8YT-0007RD-Ul; Thu, 07 Apr 2016 07:56:57 -0400\r
+Received: (nullmailer pid 4825 invoked by uid 1000);\r
+ Thu, 07 Apr 2016 11:56:46 -0000\r
+From: David Bremner <david@tethera.net>\r
+To: notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org\r
+Subject: slowdown in notmuch perf suite with xapian 1.3.5\r
+User-Agent: Notmuch/0.21+99~gd93d377 (http://notmuchmail.org) Emacs/24.5.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Thu, 07 Apr 2016 08:56:46 -0300\r
+Message-ID: <87twjd639d.fsf@zancas.localnet>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 07 Apr 2016 11:57:00 -0000\r
+\r
+\r
+I hadn't noticed any interactive slowdown, but when I got around to\r
+running the notmuch performance suite, there seems to be some noticable\r
+slowdown with the glass backend (default in Xapian 1.3.5) compared to\r
+chert (using xapian 1.2.22)\r
+\r
+These tests are on an older i7 with 12G of RAM and an SSD. I'm\r
+reasonable confident they are CPU bound. One curious thing is the\r
+increase in system time in the glass case. It also looks like the glass\r
+backend is doing a lot more I/O, which could be related.\r
+\r
+The current notmuch performance corpus has about 200k documents,\r
+totalling about 3.5G. Unfortunately each number here represents only a\r
+single run. I did rerun the tests with the glass backend, and the\r
+variation was reasonably small.\r
+\r
+Chert\r
+=====\r
+\r
+T00-new.sh: Testing notmuch new                         [0.4 large]\r
+                       Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)\r
+  Initial notmuch new   669.06 639.78  21.42   323684  3576/9360440\r
+  notmuch new #2        0.46   0.00    0.00    8240    3568/200\r
+  notmuch new #3        0.01   0.00    0.00    7916    0/8\r
+  notmuch new #4        0.01   0.01    0.00    8008    0/8\r
+  notmuch new #5        0.01   0.00    0.00    8040    0/8\r
+  notmuch new #6        0.01   0.00    0.00    8040    0/8\r
+\r
+T01-dump-restore.sh: Testing dump and restore           [0.4 large]\r
+                       Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)\r
+  load nmbug tags       5.85   2.64    0.10    11280   1376/40496\r
+  dump *                7.45   6.51    0.94    25272   104/27928\r
+  restore *             7.55   7.15    0.39    8180    0/0\r
+\r
+T02-tag.sh: Testing tagging                             [0.4 large]\r
+                       Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)\r
+  tag * +new_tag        200.13 183.01  7.08    38628   264/1664552\r
+  tag * +existing_tag   0.00   0.00    0.00    8356    0/0\r
+  tag * -existing_tag   153.47 145.00  4.02    34928   0/1626320\r
+  tag * -missing_tag    0.00   0.00    0.00    8252    0/0\r
+\r
+Glass\r
+=====\r
+\r
+T00-new.sh: Testing notmuch new                         [0.4 large]\r
+                       Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)\r
+  Initial notmuch new   949.53 697.05  206.98  277436  1290744/21767856\r
+  notmuch new #2        2.12   0.01    0.02    8204    2552/160\r
+  notmuch new #3        0.01   0.00    0.00    8216    0/8\r
+  notmuch new #4        0.01   0.00    0.00    8192    0/8\r
+  notmuch new #5        0.01   0.00    0.00    8216    0/8\r
+  notmuch new #6        0.01   0.00    0.00    8144    0/8\r
+\r
+T01-dump-restore.sh: Testing dump and restore           [0.4 large]\r
+                       Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)\r
+  load nmbug tags       10.78  4.06    3.59    11376   600/39832\r
+  dump *                7.44   6.52    0.91    25296   0/27928\r
+  restore *             7.74   7.24    0.48    8740    0/0\r
+\r
+T02-tag.sh: Testing tagging                             [0.4 large]\r
+                       Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)\r
+  tag * +new_tag        481.78 278.80  196.89  39448   0/1897360\r
+  tag * +existing_tag   0.00   0.00    0.00    8496    0/0\r
+  tag * -existing_tag   449.58 242.65  202.74  35456   0/2073520\r
+  tag * -missing_tag    0.00   0.00    0.00    8440    0/0\r
+\r
+\r