From: Austin Clements Date: Thu, 24 Jul 2014 14:32:14 +0000 (+2000) Subject: Re: Notmuch new speed degradation X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=98a6d4ffaec4554320420bb2c85aa6c1c68fa38d;p=notmuch-archives.git Re: Notmuch new speed degradation --- diff --git a/3a/ccc6b6be9e6aea449bc886ebf2689bace1e76f b/3a/ccc6b6be9e6aea449bc886ebf2689bace1e76f new file mode 100644 index 000000000..cbecc395e --- /dev/null +++ b/3a/ccc6b6be9e6aea449bc886ebf2689bace1e76f @@ -0,0 +1,100 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 19D2F431FBD + for ; Thu, 24 Jul 2014 07:32:26 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -2.3 +X-Spam-Level: +X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id heHREkDtyM3p for ; + Thu, 24 Jul 2014 07:32:22 -0700 (PDT) +Received: from dmz-mailsec-scanner-1.mit.edu (dmz-mailsec-scanner-1.mit.edu + [18.9.25.12]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 78DDB431FAF + for ; Thu, 24 Jul 2014 07:32:22 -0700 (PDT) +X-AuditID: 1209190c-f79ef6d000005dd6-01-53d118f5d638 +Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) + (using TLS with cipher AES256-SHA (256/256 bits)) + (Client did not present a certificate) + by dmz-mailsec-scanner-1.mit.edu (Symantec Messaging Gateway) with SMTP + id 83.9F.24022.5F811D35; Thu, 24 Jul 2014 10:32:21 -0400 (EDT) +Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) + by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id s6OEWJfg008537; + Thu, 24 Jul 2014 10:32:20 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s6OEWGXb032203 + (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); + Thu, 24 Jul 2014 10:32:18 -0400 +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) + (envelope-from ) + id 1XAK48-000700-0E; Thu, 24 Jul 2014 10:32:16 -0400 +Date: Thu, 24 Jul 2014 10:32:14 -0400 +From: Austin Clements +To: Dmitry Bogatov +Subject: Re: Notmuch new speed degradation +Message-ID: <20140724143214.GA13893@mit.edu> +References: <20140724081916.GA32474@localhost> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <20140724081916.GA32474@localhost> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFupjleLIzCtJLcpLzFFi42IR4hRV1v0qcTHYoOu4rsX+S19ZLK7fnMns + wOTRNs3M49mqW8wBTFFcNimpOZllqUX6dglcGSdmRRfMZqv4c2URewPjM5YuRk4OCQETiacd + 55ggbDGJC/fWs3UxcnEICcxmktjz+AuUs5FRouPtYhYI5zSTxMeJT6AySxgl/t2/xgrSzyKg + KnGjZQLYLDYBDYlt+5czgtgiAioSZ26sZgexmQWkJb79bgarERbQkph45QXYHbwCOhL7zv0E + iwsJ6Em8fnCSCSIuKHFy5hMWiF4tiRv/XgLFOcDmLP/HARLmFNCXWPayF2yVKNCqKSe3sU1g + FJqFpHsWku5ZCN0LGJlXMcqm5Fbp5iZm5hSnJusWJyfm5aUW6Rrq5WaW6KWmlG5iBAe1JM8O + xjcHlQ4xCnAwKvHwduw9HyzEmlhWXJl7iFGSg0lJlHfL2QvBQnxJ+SmVGYnFGfFFpTmpxYcY + JTiYlUR4RXkuBgvxpiRWVqUW5cOkpDlYlMR531pbBQsJpCeWpGanphakFsFkZTg4lCR4RYHR + KyRYlJqeWpGWmVOCkGbi4AQZzgM03AOkhre4IDG3ODMdIn+KUZdj0f6X3UxCLHn5ealS4rxW + IEUCIEUZpXlwc2DJ6BWjONBbwrwfxIGqeICJDG7SK6AlTEBLXiWcB1lSkoiQkmpg7F9e5Hm/ + S8os4tLZgr8RKzf3/ilZ9/WLYIzwP432ZW83Xrry4tOO/X89TyzaWXRmkWD2R+ZTVpGHPyXO + 1r3ZUrj1ovRTr89Xd2pl6NcKrJr5KPVrG9+imK78wypXfYVm3NW7aqRlY/e5mrEyZf0qVZlH + 6/yW3Lr3KejX1jViZ84rNYd/VVpvEKjEUpyRaKjFXFScCACXZLJjIQMAAA== +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 24 Jul 2014 14:32:26 -0000 + +Quoth Dmitry Bogatov on Jul 24 at 12:19 pm: +> Hello! +> +> I have ~ 3 000 000 mails. I wanted to index them. +> First 1 000 000 took several hours, next 200 000 took several days. +> +> And now, even with libeatmydata, it takes ~ 4 sec for a file. +> +> Is it any way I can improve perfomance? +> +> PS. Please, keep me in CC. + +Hi Dmitry. My guess is that's you've exceeded your OS buffer cache +size by enough that most B-tree reads are going to disk at least once. +How big is your database (du -h $MAIL/.notmuch/xapian) and what does +free -h report on that computer? Also, is this on an SSD or an HDD? + +You could try running notmuch compact. That should shrink the +database, and, more importantly, pack more into the active page set +and, I think, also linearize the database.