Re: Notmuch new speed degradation

diff --git a/3a/ccc6b6be9e6aea449bc886ebf2689bace1e76f b/3a/ccc6b6be9e6aea449bc886ebf2689bace1e76f
new file mode 100644 (file)
index 0000000..cbecc39
--- /dev/null
+++ b/3a/ccc6b6be9e6aea449bc886ebf2689bace1e76f
@@ -0,0 +1,100 @@
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 19D2F431FBD\r
+       for <notmuch@notmuchmail.org>; Thu, 24 Jul 2014 07:32:26 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id heHREkDtyM3p for <notmuch@notmuchmail.org>;\r
+       Thu, 24 Jul 2014 07:32:22 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-1.mit.edu (dmz-mailsec-scanner-1.mit.edu\r
+       [18.9.25.12])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 78DDB431FAF\r
+       for <notmuch@notmuchmail.org>; Thu, 24 Jul 2014 07:32:22 -0700 (PDT)\r
+X-AuditID: 1209190c-f79ef6d000005dd6-01-53d118f5d638\r
+Received: from mailhub-auth-1.mit.edu ( [18.9.21.35])\r
+       (using TLS with cipher AES256-SHA (256/256 bits))\r
+       (Client did not present a certificate)\r
+       by dmz-mailsec-scanner-1.mit.edu (Symantec Messaging Gateway) with SMTP\r
+       id 83.9F.24022.5F811D35; Thu, 24 Jul 2014 10:32:21 -0400 (EDT)\r
+Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
+       by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id s6OEWJfg008537; \r
+       Thu, 24 Jul 2014 10:32:20 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+       (authenticated bits=0)\r
+       (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+       by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s6OEWGXb032203\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+       Thu, 24 Jul 2014 10:32:18 -0400\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+       (envelope-from <amdragon@mit.edu>)\r
+       id 1XAK48-000700-0E; Thu, 24 Jul 2014 10:32:16 -0400\r
+Date: Thu, 24 Jul 2014 10:32:14 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Dmitry Bogatov <KAction@gnu.org>\r
+Subject: Re: Notmuch new speed degradation\r
+Message-ID: <20140724143214.GA13893@mit.edu>\r
+References: <20140724081916.GA32474@localhost>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <20140724081916.GA32474@localhost>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFupjleLIzCtJLcpLzFFi42IR4hRV1v0qcTHYoOu4rsX+S19ZLK7fnMns\r
+       wOTRNs3M49mqW8wBTFFcNimpOZllqUX6dglcGSdmRRfMZqv4c2URewPjM5YuRk4OCQETiacd\r
+       55ggbDGJC/fWs3UxcnEICcxmktjz+AuUs5FRouPtYhYI5zSTxMeJT6AySxgl/t2/xgrSzyKg\r
+       KnGjZQLYLDYBDYlt+5czgtgiAioSZ26sZgexmQWkJb79bgarERbQkph45QXYHbwCOhL7zv0E\r
+       iwsJ6Em8fnCSCSIuKHFy5hMWiF4tiRv/XgLFOcDmLP/HARLmFNCXWPayF2yVKNCqKSe3sU1g\r
+       FJqFpHsWku5ZCN0LGJlXMcqm5Fbp5iZm5hSnJusWJyfm5aUW6Rrq5WaW6KWmlG5iBAe1JM8O\r
+       xjcHlQ4xCnAwKvHwduw9HyzEmlhWXJl7iFGSg0lJlHfL2QvBQnxJ+SmVGYnFGfFFpTmpxYcY\r
+       JTiYlUR4RXkuBgvxpiRWVqUW5cOkpDlYlMR531pbBQsJpCeWpGanphakFsFkZTg4lCR4RYHR\r
+       KyRYlJqeWpGWmVOCkGbi4AQZzgM03AOkhre4IDG3ODMdIn+KUZdj0f6X3UxCLHn5ealS4rxW\r
+       IEUCIEUZpXlwc2DJ6BWjONBbwrwfxIGqeICJDG7SK6AlTEBLXiWcB1lSkoiQkmpg7F9e5Hm/\r
+       S8os4tLZgr8RKzf3/ilZ9/WLYIzwP432ZW83Xrry4tOO/X89TyzaWXRmkWD2R+ZTVpGHPyXO\r
+       1r3ZUrj1ovRTr89Xd2pl6NcKrJr5KPVrG9+imK78wypXfYVm3NW7aqRlY/e5mrEyZf0qVZlH\r
+       6/yW3Lr3KejX1jViZ84rNYd/VVpvEKjEUpyRaKjFXFScCACXZLJjIQMAAA==\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 24 Jul 2014 14:32:26 -0000\r
+\r
+Quoth Dmitry Bogatov on Jul 24 at 12:19 pm:\r
+> Hello!\r
+> \r
+> I have ~ 3 000 000 mails. I wanted to index them.\r
+> First 1 000 000 took several hours, next 200 000 took several days.\r
+> \r
+> And now, even with libeatmydata, it takes ~ 4 sec for a file.\r
+> \r
+> Is it any way I can improve perfomance?\r
+> \r
+> PS. Please, keep me in CC.\r
+\r
+Hi Dmitry.  My guess is that's you've exceeded your OS buffer cache\r
+size by enough that most B-tree reads are going to disk at least once.\r
+How big is your database (du -h $MAIL/.notmuch/xapian) and what does\r
+free -h report on that computer?  Also, is this on an SSD or an HDD?\r
+\r
+You could try running notmuch compact.  That should shrink the\r
+database, and, more importantly, pack more into the active page set\r
+and, I think, also linearize the database.\r