Re: slowdown in notmuch perf suite with xapian 1.3.5

diff --git a/7f/65fbd68ff0dd697144fb8651add28d9572b22f b/7f/65fbd68ff0dd697144fb8651add28d9572b22f
new file mode 100644 (file)
index 0000000..6efc8cc
--- /dev/null
+++ b/7f/65fbd68ff0dd697144fb8651add28d9572b22f
@@ -0,0 +1,101 @@
+Return-Path: <olly@survex.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 5D4F26DE02B5\r
+ for <notmuch@notmuchmail.org>; Thu,  7 Apr 2016 17:57:35 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.377\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.377 tagged_above=-999 required=5\r
+ tests=[AWL=-0.076, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001]\r
+ autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 7JG4OhwXFuzJ for <notmuch@notmuchmail.org>;\r
+ Thu,  7 Apr 2016 17:57:27 -0700 (PDT)\r
+Received: from atreus.tartarus.org (atreus.tartarus.org [80.252.125.10])\r
+ by arlo.cworth.org (Postfix) with ESMTPS id 8387A6DE0134\r
+ for <notmuch@notmuchmail.org>; Thu,  7 Apr 2016 17:57:27 -0700 (PDT)\r
+Received: from olly by atreus.tartarus.org with local (Exim 4.69)\r
+ (envelope-from <olly@survex.com>)\r
+ id 1aoKjl-0001nA-D6; Fri, 08 Apr 2016 01:57:25 +0100\r
+Date: Fri, 8 Apr 2016 01:57:25 +0100\r
+From: Olly Betts <olly@survex.com>\r
+To: David Bremner <david@tethera.net>\r
+Cc: notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org\r
+Subject: Re: slowdown in notmuch perf suite with xapian 1.3.5\r
+Message-ID: <20160408005725.GA3037@survex.com>\r
+Reply-To: Xapian Discussion <xapian-discuss@lists.xapian.org>\r
+Mail-Followup-To: David Bremner <david@tethera.net>,\r
+ notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org\r
+References: <87twjd639d.fsf@zancas.localnet>\r
+ <20160407232537.GB29434@survex.com>\r
+ <87h9fd53vo.fsf@zancas.localnet>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <87h9fd53vo.fsf@zancas.localnet>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 08 Apr 2016 00:57:35 -0000\r
+\r
+On Thu, Apr 07, 2016 at 09:40:59PM -0300, David Bremner wrote:\r
+> Olly Betts <olly@survex.com> writes:\r
+> \r
+> >\r
+> > So the T00-new.sh numbers make sense - there's more work to do, and\r
+> > we need to read existing positional data more to insert the new stuff,\r
+> > so the increased reads and writes make sense.\r
+> >\r
+> > But guessing at what the other two tests do, I wouldn't expect them to\r
+> > be affected by this.\r
+> \r
+> The non-optimized-away cases of T02-tag just adding and deleting terms\r
+> to each document with term Tmail\r
+\r
+That should short-cut to just only changing the data for Tmail.  Perhaps that's\r
+not working correctly - I'll take a look at this, but probably after 1.4.0 is\r
+out.\r
+\r
+> > I'm also a bit puzzled by how glass can manage not to read any data\r
+> > for "dump *", and several tests seem to not read or write anything\r
+> > for either backend.  What exactly are the "In/Out" numbers?\r
+> \r
+> that's just the output from /usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O'\r
+> \r
+> The manual describes them as "number of file system\r
+> inputs/outputs". From looking at the source, they correspond to\r
+> ru_inblock and ru_oublock fields from the getrusage call. AFAIU, that\r
+> means the number of non-cached read/writes.\r
+\r
+Non-cached reads/writes are arguably the most useful sort to measure, but the\r
+reads at least will be sensitive to OS caching, which means a repeat run will\r
+generally show lower numbers of reads, e.g.:\r
+\r
+$ /usr/bin/time -f '%I/%O' wc randomfile \r
+  240  2908 96780 randomfile\r
+192/0\r
+$ /usr/bin/time -f '%I/%O' wc randomfile \r
+  240  2908 96780 randomfile\r
+0/0\r
+\r
+So those numbers may not be entirely comparable, depending what order your\r
+tests were done in, and whether you'd run the tests (or cloned the repo or some\r
+other operation which read or wrote the files used) recently enough that their\r
+data might still be cached.\r
+\r
+Cheers,\r
+    Olly\r