Re: [PATCH v2] new: Don't scan unchanged directories with no sub-directories

diff --git a/d0/82d2c2c80e8237b8adf04727d65cf13a69d4ad b/d0/82d2c2c80e8237b8adf04727d65cf13a69d4ad
new file mode 100644 (file)
index 0000000..722d385
--- /dev/null
+++ b/d0/82d2c2c80e8237b8adf04727d65cf13a69d4ad
@@ -0,0 +1,118 @@
+Return-Path: <tomi.ollila@iki.fi>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 61830431FC2\r
+       for <notmuch@notmuchmail.org>; Fri, 25 Oct 2013 04:46:39 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id u38zQyKnXA4L for <notmuch@notmuchmail.org>;\r
+       Fri, 25 Oct 2013 04:46:31 -0700 (PDT)\r
+Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])\r
+       by olra.theworths.org (Postfix) with ESMTP id CF22C431FB6\r
+       for <notmuch@notmuchmail.org>; Fri, 25 Oct 2013 04:46:30 -0700 (PDT)\r
+Received: from guru.guru-group.fi (localhost [IPv6:::1])\r
+       by guru.guru-group.fi (Postfix) with ESMTP id 364EB100217;\r
+       Fri, 25 Oct 2013 14:46:21 +0300 (EEST)\r
+From: Tomi Ollila <tomi.ollila@iki.fi>\r
+To: Austin Clements <amdragon@MIT.EDU>, notmuch@notmuchmail.org\r
+Subject: Re: [PATCH v2] new: Don't scan unchanged directories with no\r
+       sub-directories\r
+In-Reply-To: <1382650739-12438-1-git-send-email-amdragon@mit.edu>\r
+References: <20131024210837.GH20337@mit.edu>\r
+       <1382650739-12438-1-git-send-email-amdragon@mit.edu>\r
+User-Agent: Notmuch/0.16+115~g11c2ff5 (http://notmuchmail.org) Emacs/24.3.1\r
+       (x86_64-unknown-linux-gnu)\r
+X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL\r
+       $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F\r
+       !)g;OY^,BjTbr)Np:%c_o'jj,Z\r
+Date: Fri, 25 Oct 2013 14:46:21 +0300\r
+Message-ID: <m28uxhefoy.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 25 Oct 2013 11:46:39 -0000\r
+\r
+On Fri, Oct 25 2013, Austin Clements <amdragon@MIT.EDU> wrote:\r
+\r
+> This can substantially reduce the cost of notmuch new in some\r
+> situations, such as when the file system cache is cold or when the\r
+> Maildir is on NFS.\r
+> ---\r
+\r
+LGTM. The creation and destruction of child directories happens\r
+only if there are symlinks to directories in otherwise leaf directories.\r
+\r
+Tomi\r
+\r
+>\r
+> This should fix the problem with directories containing symlinks to\r
+> other directories, but no actual sub-directories.\r
+>\r
+>  notmuch-new.c | 29 +++++++++++++++++++++++++++++\r
+>  1 file changed, 29 insertions(+)\r
+>\r
+> diff --git a/notmuch-new.c b/notmuch-new.c\r
+> index faa33f1..ba05cb4 100644\r
+> --- a/notmuch-new.c\r
+> +++ b/notmuch-new.c\r
+> @@ -323,6 +323,35 @@ add_files (notmuch_database_t *notmuch,\r
+>      }\r
+>      db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;\r
+>  \r
+> +    /* If the directory is unchanged from our last scan and has no\r
+> +     * sub-directories, then return without scanning it at all.  In\r
+> +     * some situations, skipping the scan can substantially reduce the\r
+> +     * cost of notmuch new, especially since the huge numbers of files\r
+> +     * in Maildirs make scans expensive, but all files live in leaf\r
+> +     * directories.\r
+> +     *\r
+> +     * To check for sub-directories, we borrow a trick from find,\r
+> +     * kpathsea, and many other UNIX tools: since a directory's link\r
+> +     * count is the number of sub-directories (specifically, their\r
+> +     * '..' entries) plus 2 (the link from the parent and the link for\r
+> +     * '.').  This check is safe even on weird file systems, since\r
+> +     * file systems that can't compute this will return 0 or 1.  This\r
+> +     * is safe even on *really* weird file systems like HFS+ that\r
+> +     * mistakenly return the total number of directory entries, since\r
+> +     * that only inflates the count beyond 2.\r
+> +     */\r
+> +    if (directory && fs_mtime == db_mtime && st.st_nlink == 2) {\r
+> +    /* There's one catch: pass 1 below considers symlinks to\r
+> +     * directories to be directories, but these don't increase the\r
+> +     * file system link count.  So, only bail early if the\r
+> +     * database agrees that there are no sub-directories. */\r
+> +    db_subdirs = notmuch_directory_get_child_directories (directory);\r
+> +    if (!notmuch_filenames_valid (db_subdirs))\r
+> +        goto DONE;\r
+> +    notmuch_filenames_destroy (db_subdirs);\r
+> +    db_subdirs = NULL;\r
+> +    }\r
+> +\r
+>      /* If the database knows about this directory, then we sort based\r
+>       * on strcmp to match the database sorting. Otherwise, we can do\r
+>       * inode-based sorting for faster filesystem operation. */\r
+> -- \r
+> 1.8.4.rc3\r
+>\r
+> _______________________________________________\r
+> notmuch mailing list\r
+> notmuch@notmuchmail.org\r
+> http://notmuchmail.org/mailman/listinfo/notmuch\r