--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 3E170431FBD\r
+ for <notmuch@notmuchmail.org>; Thu, 24 Jul 2014 15:32:00 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id fIk1XAhiRXKF for <notmuch@notmuchmail.org>;\r
+ Thu, 24 Jul 2014 15:31:56 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-2.mit.edu (dmz-mailsec-scanner-2.mit.edu\r
+ [18.9.25.13])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id D014F431FAF\r
+ for <notmuch@notmuchmail.org>; Thu, 24 Jul 2014 15:31:55 -0700 (PDT)\r
+X-AuditID: 1209190d-f79c06d000002f07-fe-53d1895bfd75\r
+Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
+ (using TLS with cipher AES256-SHA (256/256 bits))\r
+ (Client did not present a certificate)\r
+ by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 1F.56.12039.B5981D35; Thu, 24 Jul 2014 18:31:55 -0400 (EDT)\r
+Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
+ by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id s6OMVrFI029308; \r
+ Thu, 24 Jul 2014 18:31:54 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s6OMVpXd001562\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+ Thu, 24 Jul 2014 18:31:53 -0400\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+ (envelope-from <amdragon@MIT.EDU>)\r
+ id 1XARYB-0001Q5-FW; Thu, 24 Jul 2014 18:31:50 -0400\r
+Date: Thu, 24 Jul 2014 18:31:45 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Dmitry Bogatov <KAction@gnu.org>\r
+Subject: Re: Notmuch new speed degradation\r
+Message-ID: <20140724223145.GB13893@mit.edu>\r
+References: <20140724081916.GA32474@localhost>\r
+ <20140724143214.GA13893@mit.edu> <20140724194946.GA4724@localhost>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <20140724194946.GA4724@localhost>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFmpkleLIzCtJLcpLzFFi42IR4hTV1o3uvBhsMLWH32L/pa8sFtdvzmR2\r
+ YPJom2bm8WzVLeYApigum5TUnMyy1CJ9uwSujLtbd7AVTBOsWLnkI3MDYxtvFyMHh4SAicTH\r
+ u6xdjJxAppjEhXvr2boYuTiEBGYzSSyc+IUdwtnIKPGhZSZU5jSTxPxj58FahASWMEps+2kA\r
+ YrMIqEq8fXubCcRmE9CQ2LZ/OSOILSKgInHmxmp2EJtZQFri2+9msBphAS2JiVdesIDYvAI6\r
+ Equ3fGKHmFkqser3dGaIuKDEyZlPWCB6tSRu/HvJBHI1yJzl/zhAwpwCehLzJnwGGykKtGrK\r
+ yW1sExiFZiHpnoWkexZC9wJG5lWMsim5Vbq5iZk5xanJusXJiXl5qUW6Rnq5mSV6qSmlmxjB\r
+ IS3Ju4Px3UGlQ4wCHIxKPLwd9ReDhVgTy4orcw8xSnIwKYnyTm8BCvEl5adUZiQWZ8QXleak\r
+ Fh9ilOBgVhLh3RgElONNSaysSi3Kh0lJc7AoifO+tbYKFhJITyxJzU5NLUgtgsnKcHAoSfB+\r
+ awdqFCxKTU+tSMvMKUFIM3FwggznARou0AEyvLggMbc4Mx0if4pRUUqc1x6kWQAkkVGaB9cL\r
+ SzmvGMWBXhHmZQRp5wGmK7juV0CDmYAGv0o4DzK4JBEhJdXAmMYldHDJUladNRG+79LcxPQq\r
+ /j+KWrKk4NRMqcm3g05xc7KdUJEQfiDznPHkt5uR11WWVtkI+tX4d9QGKF5afll+xgaP3DWJ\r
+ TqsK70YKs73sXSPc/C7y8i+BH90c8T5+/o23ljzXk5WJvnlE6+Sp+PaCZS6WS1zYJXfeuqe0\r
+ 5Twro9YukexrSizFGYmGWsxFxYkAE3H5FRQDAAA=\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 24 Jul 2014 22:32:00 -0000\r
+\r
+Quoth Dmitry Bogatov on Jul 24 at 11:49 pm:\r
+> * Austin Clements <amdragon@MIT.EDU> [2014-07-24 10:32:14-0400]\r
+> > Hi Dmitry. My guess is that's you've exceeded your OS buffer cache\r
+> > size by enough that most B-tree reads are going to disk at least once.\r
+> > How big is your database (du -h $MAIL/.notmuch/xapian) and what does\r
+> > free -h report on that computer? Also, is this on an SSD or an HDD?\r
+> \r
+> 13Gb on HDD, 9G after compact. Compact did not improved indexing speed,\r
+> unfortunately. Maybe it is possible to somehow merge databases?\r
+\r
+Unfortunately, there's no support for merging databases. Other than\r
+technical difficulties like identifying messages that should belong to\r
+the same thread during merge, the schema wasn't designed with this in\r
+mind and uses various features that are incompatible with merging.\r
+\r
+There are some known problems with Xapian slowing down as the database\r
+gets larger, but four seconds per message still sounds extreme.\r
+\r
+Another thing to try is to raise Xapian's flush threshold by setting\r
+the environment variable XAPIAN_FLUSH_THRESHOLD. The default is\r
+10000. Try increasing it by, say, an order of magnitude (you can\r
+probably go much higher than that, though you don't want to go too\r
+high or you'll start eating in to the memory for your page cache).\r
+\r
+> total used free shared buffers cached\r
+> Mem: 7,7G 6,5G 1,2G 240M 826M 3,6G\r
+> -/+ buffers/cache: 2,1G 5,6G\r
+> Swap: 1,9G 66M 1,8G\r
+\r
+Hmm. Was this after the compact or after notmuch new had run for a\r
+while? 1.2GB of free memory suggests that it's not a page cache\r
+problem, but that would only apply if you took this snapshot after\r
+notmuch new, not after compact.\r
+\r
+We should confirm that this is an IO problem. If you run\r
+/usr/bin/time notmuch new for a few minutes, is the %CPU significantly\r
+below 100%? If it's above 90%ish, then this is a CPU problem and we\r
+might be able to track it down using CPU profiling. If it is an IO\r
+problem (which is almost certainly is), I'm afraid it's much harder to\r
+track down.\r
+\r
+Also, what file system are you using?\r
projects / notmuch-archives.git / commitdiff