From e2bcc8c311e9cea737ab9673b25d524a44b239a8 Mon Sep 17 00:00:00 2001 From: Sascha Silbe Date: Sun, 24 Jun 2012 18:29:23 +0200 Subject: [PATCH] [PATCH 0/3] Speed up notmuch new for unchanged directories --- e5/c247ca37b58fcc6f90804ad3d2ffa303ba17e7 | 75 +++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 e5/c247ca37b58fcc6f90804ad3d2ffa303ba17e7 diff --git a/e5/c247ca37b58fcc6f90804ad3d2ffa303ba17e7 b/e5/c247ca37b58fcc6f90804ad3d2ffa303ba17e7 new file mode 100644 index 000000000..a46f7ab13 --- /dev/null +++ b/e5/c247ca37b58fcc6f90804ad3d2ffa303ba17e7 @@ -0,0 +1,75 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 978E4431FAF + for ; Sun, 24 Jun 2012 09:29:43 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id T0rOlydDZeIF for ; + Sun, 24 Jun 2012 09:29:43 -0700 (PDT) +Received: from smtp.chost.de (setoy.chost.de [217.160.209.225]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id A79AA431FAE + for ; Sun, 24 Jun 2012 09:29:42 -0700 (PDT) +Received: (qmail 14457 invoked by uid 5015); 24 Jun 2012 16:29:40 -0000 +Received: (nullmailer pid 3401 invoked by uid 123); + Sun, 24 Jun 2012 16:29:39 -0000 +Received: from twin.sascha.silbe.org (twin.sascha.silbe.org [192.168.1.2]) + by flatty.sascha.silbe.org ([192.168.1.252]) + with SMTP via TCP; 24 Jun 2012 16:29:39 -0000 +Received: (nullmailer pid 25943 invoked by uid 8193); + Sun, 24 Jun 2012 16:29:39 -0000 +From: Sascha Silbe +To: notmuch +Subject: [PATCH 0/3] Speed up notmuch new for unchanged directories +Date: Sun, 24 Jun 2012 18:29:23 +0200 +Message-Id: <1340555366-25891-1-git-send-email-sascha-pgp@silbe.org> +X-Mailer: git-send-email 1.7.10 +Mail-Followup-To: notmuch +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +Reply-To: Sascha Silbe +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 24 Jun 2012 16:29:43 -0000 + +All the time I thought what makes "notmuch new" so abysmally slow is the +stat() for each maildir. But as it continued to be slow even after I +moved most mails out of 'new' (into 'new-20120624'), I strace'd notmuch +and noticed it listed even unchanged directories, thereby listing and +iterating over each and every single of the 900k mails in my mail store. + +There's still quite some room for further improvements as it continues +to take several minutes to scan < 100 new mails in changed directories +containing < 1000 mails in total. Even the rsync run that fetches the +new mails is faster. + +Sascha Silbe (3): + lib: fix NULL checks for filenames iterators + lib: add support for rewinding a filenames iterator + new: don't read unchanged directories from disk + + lib/filenames.c | 15 ++++++- + lib/notmuch.h | 8 ++++ + notmuch-new.c | 130 +++++++++++++++++++++++++++++++++++++++++-------------- + 3 files changed, 119 insertions(+), 34 deletions(-) + +-- +1.7.10 + -- 2.26.2