From: Austin Clements Date: Thu, 24 Jul 2014 22:31:45 +0000 (+2000) Subject: Re: Notmuch new speed degradation X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=592e0a310d224135cede8abbb92d5d6e5d901bc5;p=notmuch-archives.git Re: Notmuch new speed degradation --- diff --git a/53/d90a5b352c77b1abc6cf6f8e4bc4f6c5aaca06 b/53/d90a5b352c77b1abc6cf6f8e4bc4f6c5aaca06 new file mode 100644 index 000000000..75fa43f93 --- /dev/null +++ b/53/d90a5b352c77b1abc6cf6f8e4bc4f6c5aaca06 @@ -0,0 +1,123 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 3E170431FBD + for ; Thu, 24 Jul 2014 15:32:00 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -2.3 +X-Spam-Level: +X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id fIk1XAhiRXKF for ; + Thu, 24 Jul 2014 15:31:56 -0700 (PDT) +Received: from dmz-mailsec-scanner-2.mit.edu (dmz-mailsec-scanner-2.mit.edu + [18.9.25.13]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id D014F431FAF + for ; Thu, 24 Jul 2014 15:31:55 -0700 (PDT) +X-AuditID: 1209190d-f79c06d000002f07-fe-53d1895bfd75 +Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) + (using TLS with cipher AES256-SHA (256/256 bits)) + (Client did not present a certificate) + by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP + id 1F.56.12039.B5981D35; Thu, 24 Jul 2014 18:31:55 -0400 (EDT) +Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) + by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id s6OMVrFI029308; + Thu, 24 Jul 2014 18:31:54 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s6OMVpXd001562 + (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); + Thu, 24 Jul 2014 18:31:53 -0400 +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) + (envelope-from ) + id 1XARYB-0001Q5-FW; Thu, 24 Jul 2014 18:31:50 -0400 +Date: Thu, 24 Jul 2014 18:31:45 -0400 +From: Austin Clements +To: Dmitry Bogatov +Subject: Re: Notmuch new speed degradation +Message-ID: <20140724223145.GB13893@mit.edu> +References: <20140724081916.GA32474@localhost> + <20140724143214.GA13893@mit.edu> <20140724194946.GA4724@localhost> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <20140724194946.GA4724@localhost> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFmpkleLIzCtJLcpLzFFi42IR4hTV1o3uvBhsMLWH32L/pa8sFtdvzmR2 + YPJom2bm8WzVLeYApigum5TUnMyy1CJ9uwSujLtbd7AVTBOsWLnkI3MDYxtvFyMHh4SAicTH + u6xdjJxAppjEhXvr2boYuTiEBGYzSSyc+IUdwtnIKPGhZSZU5jSTxPxj58FahASWMEps+2kA + YrMIqEq8fXubCcRmE9CQ2LZ/OSOILSKgInHmxmp2EJtZQFri2+9msBphAS2JiVdesIDYvAI6 + Equ3fGKHmFkqser3dGaIuKDEyZlPWCB6tSRu/HvJBHI1yJzl/zhAwpwCehLzJnwGGykKtGrK + yW1sExiFZiHpnoWkexZC9wJG5lWMsim5Vbq5iZk5xanJusXJiXl5qUW6Rnq5mSV6qSmlmxjB + IS3Ju4Px3UGlQ4wCHIxKPLwd9ReDhVgTy4orcw8xSnIwKYnyTm8BCvEl5adUZiQWZ8QXleak + Fh9ilOBgVhLh3RgElONNSaysSi3Kh0lJc7AoifO+tbYKFhJITyxJzU5NLUgtgsnKcHAoSfB+ + awdqFCxKTU+tSMvMKUFIM3FwggznARou0AEyvLggMbc4Mx0if4pRUUqc1x6kWQAkkVGaB9cL + SzmvGMWBXhHmZQRp5wGmK7juV0CDmYAGv0o4DzK4JBEhJdXAmMYldHDJUladNRG+79LcxPQq + /j+KWrKk4NRMqcm3g05xc7KdUJEQfiDznPHkt5uR11WWVtkI+tX4d9QGKF5afll+xgaP3DWJ + TqsK70YKs73sXSPc/C7y8i+BH90c8T5+/o23ljzXk5WJvnlE6+Sp+PaCZS6WS1zYJXfeuqe0 + 5Twro9YukexrSizFGYmGWsxFxYkAE3H5FRQDAAA= +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 24 Jul 2014 22:32:00 -0000 + +Quoth Dmitry Bogatov on Jul 24 at 11:49 pm: +> * Austin Clements [2014-07-24 10:32:14-0400] +> > Hi Dmitry. My guess is that's you've exceeded your OS buffer cache +> > size by enough that most B-tree reads are going to disk at least once. +> > How big is your database (du -h $MAIL/.notmuch/xapian) and what does +> > free -h report on that computer? Also, is this on an SSD or an HDD? +> +> 13Gb on HDD, 9G after compact. Compact did not improved indexing speed, +> unfortunately. Maybe it is possible to somehow merge databases? + +Unfortunately, there's no support for merging databases. Other than +technical difficulties like identifying messages that should belong to +the same thread during merge, the schema wasn't designed with this in +mind and uses various features that are incompatible with merging. + +There are some known problems with Xapian slowing down as the database +gets larger, but four seconds per message still sounds extreme. + +Another thing to try is to raise Xapian's flush threshold by setting +the environment variable XAPIAN_FLUSH_THRESHOLD. The default is +10000. Try increasing it by, say, an order of magnitude (you can +probably go much higher than that, though you don't want to go too +high or you'll start eating in to the memory for your page cache). + +> total used free shared buffers cached +> Mem: 7,7G 6,5G 1,2G 240M 826M 3,6G +> -/+ buffers/cache: 2,1G 5,6G +> Swap: 1,9G 66M 1,8G + +Hmm. Was this after the compact or after notmuch new had run for a +while? 1.2GB of free memory suggests that it's not a page cache +problem, but that would only apply if you took this snapshot after +notmuch new, not after compact. + +We should confirm that this is an IO problem. If you run +/usr/bin/time notmuch new for a few minutes, is the %CPU significantly +below 100%? If it's above 90%ish, then this is a CPU problem and we +might be able to track it down using CPU profiling. If it is an IO +problem (which is almost certainly is), I'm afraid it's much harder to +track down. + +Also, what file system are you using?