Re: slowdown in notmuch perf suite with xapian 1.3.5

diff --git a/ca/aae1e2b616d9d51c73c1318f5a311cba3c66b6 b/ca/aae1e2b616d9d51c73c1318f5a311cba3c66b6
new file mode 100644 (file)
index 0000000..7689643
--- /dev/null
+++ b/ca/aae1e2b616d9d51c73c1318f5a311cba3c66b6
@@ -0,0 +1,91 @@
+Return-Path: <olly@survex.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 23BCB6DE02B5\r
+ for <notmuch@notmuchmail.org>; Thu,  7 Apr 2016 17:03:15 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.421\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.421 tagged_above=-999 required=5\r
+ tests=[AWL=-0.120, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001]\r
+ autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id zCKOoGFKIfci for <notmuch@notmuchmail.org>;\r
+ Thu,  7 Apr 2016 17:03:07 -0700 (PDT)\r
+X-Greylist: delayed 2244 seconds by postgrey-1.35 at arlo;\r
+ Thu, 07 Apr 2016 17:03:06 PDT\r
+Received: from atreus.tartarus.org (atreus.tartarus.org [80.252.125.10])\r
+ by arlo.cworth.org (Postfix) with ESMTPS id CF23C6DE0134\r
+ for <notmuch@notmuchmail.org>; Thu,  7 Apr 2016 17:03:06 -0700 (PDT)\r
+Received: from olly by atreus.tartarus.org with local (Exim 4.69)\r
+ (envelope-from <olly@survex.com>)\r
+ id 1aoJIw-0002w0-2y; Fri, 08 Apr 2016 00:25:38 +0100\r
+Date: Fri, 8 Apr 2016 00:25:38 +0100\r
+From: Olly Betts <olly@survex.com>\r
+To: David Bremner <david@tethera.net>\r
+Cc: notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org\r
+Subject: Re: slowdown in notmuch perf suite with xapian 1.3.5\r
+Message-ID: <20160407232537.GB29434@survex.com>\r
+Reply-To: Xapian Discussion <xapian-discuss@lists.xapian.org>\r
+Mail-Followup-To: David Bremner <david@tethera.net>,\r
+ notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org\r
+References: <87twjd639d.fsf@zancas.localnet>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <87twjd639d.fsf@zancas.localnet>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 08 Apr 2016 00:03:15 -0000\r
+\r
+On Thu, Apr 07, 2016 at 08:56:46AM -0300, David Bremner wrote:\r
+> I hadn't noticed any interactive slowdown, but when I got around to\r
+> running the notmuch performance suite, there seems to be some noticable\r
+> slowdown with the glass backend (default in Xapian 1.3.5) compared to\r
+> chert (using xapian 1.2.22)\r
+\r
+Some of this is pretty much expected, though other parts I don't\r
+entirely understand.\r
+\r
+One of the big changes in glass is how the position table is structured.\r
+In chert, it is ordered by (document,term) but in glass that has been\r
+changed to (term,document).\r
+\r
+This change makes a huge difference to phrase searches in cases where\r
+a lot of phrase data is needed, but it has an indexing time cost -\r
+adding a new document can no longer just append a load of entries to\r
+the position table, but instead we need to buffer up the changes, and\r
+then merge the entries within the existing table.\r
+\r
+The trade-off isn't ideal for everyone, but the cases of slow phrase\r
+searches were a real pain point that needed addressing.  The plan is\r
+to optimise indexing speed in other ways to regain this loss - some\r
+of that has been done but there's a lot more to do still.\r
+\r
+So the T00-new.sh numbers make sense - there's more work to do, and\r
+we need to read existing positional data more to insert the new stuff,\r
+so the increased reads and writes make sense.\r
+\r
+But guessing at what the other two tests do, I wouldn't expect them to\r
+be affected by this.\r
+\r
+I'm also a bit puzzled by how glass can manage not to read any data\r
+for "dump *", and several tests seem to not read or write anything\r
+for either backend.  What exactly are the "In/Out" numbers?\r
+\r
+Cheers,\r
+    Olly\r