Return-Path: X-Original-To: notmuch@notmuchmail.org Delivered-To: notmuch@notmuchmail.org Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 19D2F431FBD for ; Thu, 24 Jul 2014 07:32:26 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -2.3 X-Spam-Level: X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id heHREkDtyM3p for ; Thu, 24 Jul 2014 07:32:22 -0700 (PDT) Received: from dmz-mailsec-scanner-1.mit.edu (dmz-mailsec-scanner-1.mit.edu [18.9.25.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 78DDB431FAF for ; Thu, 24 Jul 2014 07:32:22 -0700 (PDT) X-AuditID: 1209190c-f79ef6d000005dd6-01-53d118f5d638 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-1.mit.edu (Symantec Messaging Gateway) with SMTP id 83.9F.24022.5F811D35; Thu, 24 Jul 2014 10:32:21 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id s6OEWJfg008537; Thu, 24 Jul 2014 10:32:20 -0400 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) (authenticated bits=0) (User authenticated as amdragon@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s6OEWGXb032203 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 24 Jul 2014 10:32:18 -0400 Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) (envelope-from ) id 1XAK48-000700-0E; Thu, 24 Jul 2014 10:32:16 -0400 Date: Thu, 24 Jul 2014 10:32:14 -0400 From: Austin Clements To: Dmitry Bogatov Subject: Re: Notmuch new speed degradation Message-ID: <20140724143214.GA13893@mit.edu> References: <20140724081916.GA32474@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140724081916.GA32474@localhost> User-Agent: Mutt/1.5.21 (2010-09-15) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupjleLIzCtJLcpLzFFi42IR4hRV1v0qcTHYoOu4rsX+S19ZLK7fnMns wOTRNs3M49mqW8wBTFFcNimpOZllqUX6dglcGSdmRRfMZqv4c2URewPjM5YuRk4OCQETiacd 55ggbDGJC/fWs3UxcnEICcxmktjz+AuUs5FRouPtYhYI5zSTxMeJT6AySxgl/t2/xgrSzyKg KnGjZQLYLDYBDYlt+5czgtgiAioSZ26sZgexmQWkJb79bgarERbQkph45QXYHbwCOhL7zv0E iwsJ6Em8fnCSCSIuKHFy5hMWiF4tiRv/XgLFOcDmLP/HARLmFNCXWPayF2yVKNCqKSe3sU1g FJqFpHsWku5ZCN0LGJlXMcqm5Fbp5iZm5hSnJusWJyfm5aUW6Rrq5WaW6KWmlG5iBAe1JM8O xjcHlQ4xCnAwKvHwduw9HyzEmlhWXJl7iFGSg0lJlHfL2QvBQnxJ+SmVGYnFGfFFpTmpxYcY JTiYlUR4RXkuBgvxpiRWVqUW5cOkpDlYlMR531pbBQsJpCeWpGanphakFsFkZTg4lCR4RYHR KyRYlJqeWpGWmVOCkGbi4AQZzgM03AOkhre4IDG3ODMdIn+KUZdj0f6X3UxCLHn5ealS4rxW IEUCIEUZpXlwc2DJ6BWjONBbwrwfxIGqeICJDG7SK6AlTEBLXiWcB1lSkoiQkmpg7F9e5Hm/ S8os4tLZgr8RKzf3/ilZ9/WLYIzwP432ZW83Xrry4tOO/X89TyzaWXRmkWD2R+ZTVpGHPyXO 1r3ZUrj1ovRTr89Xd2pl6NcKrJr5KPVrG9+imK78wypXfYVm3NW7aqRlY/e5mrEyZf0qVZlH 6/yW3Lr3KejX1jViZ84rNYd/VVpvEKjEUpyRaKjFXFScCACXZLJjIQMAAA== Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 14:32:26 -0000 Quoth Dmitry Bogatov on Jul 24 at 12:19 pm: > Hello! > > I have ~ 3 000 000 mails. I wanted to index them. > First 1 000 000 took several hours, next 200 000 took several days. > > And now, even with libeatmydata, it takes ~ 4 sec for a file. > > Is it any way I can improve perfomance? > > PS. Please, keep me in CC. Hi Dmitry. My guess is that's you've exceeded your OS buffer cache size by enough that most B-tree reads are going to disk at least once. How big is your database (du -h $MAIL/.notmuch/xapian) and what does free -h report on that computer? Also, is this on an SSD or an HDD? You could try running notmuch compact. That should shrink the database, and, more importantly, pack more into the active page set and, I think, also linearize the database.