Re: [PATCH 0/3] Speed up notmuch new for unchanged directories

diff --git a/8f/0251b391b15382a4c4ec739819a590af50128b b/8f/0251b391b15382a4c4ec739819a590af50128b
new file mode 100644 (file)
index 0000000..62219b8
--- /dev/null
+++ b/8f/0251b391b15382a4c4ec739819a590af50128b
@@ -0,0 +1,158 @@
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 9B42A431FB6\r
+       for <notmuch@notmuchmail.org>; Mon, 25 Jun 2012 18:48:00 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id uS-oFf0wmzln for <notmuch@notmuchmail.org>;\r
+       Mon, 25 Jun 2012 18:47:58 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-3.mit.edu (DMZ-MAILSEC-SCANNER-3.MIT.EDU\r
+       [18.9.25.14])\r
+       by olra.theworths.org (Postfix) with ESMTP id 44F36431FAF\r
+       for <notmuch@notmuchmail.org>; Mon, 25 Jun 2012 18:47:58 -0700 (PDT)\r
+X-AuditID: 1209190e-b7fb56d0000008b2-0c-4fe914cc9f7c\r
+Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])\r
+       by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP\r
+       id F7.2C.02226.CC419EF4; Mon, 25 Jun 2012 21:47:56 -0400 (EDT)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+       by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id q5Q1ltJq011643; \r
+       Mon, 25 Jun 2012 21:47:56 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+       (authenticated bits=0)\r
+       (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+       by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q5Q1lsP2026639\r
+       (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+       Mon, 25 Jun 2012 21:47:55 -0400 (EDT)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
+       (envelope-from <amdragon@mit.edu>)\r
+       id 1SjKsk-0005uu-9E; Mon, 25 Jun 2012 21:47:54 -0400\r
+Date: Mon, 25 Jun 2012 21:47:54 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Sascha Silbe <sascha-ml-reply-to-2012-3@silbe.org>\r
+Subject: Re: [PATCH 0/3] Speed up notmuch new for unchanged directories\r
+Message-ID: <20120626014754.GO24342@mit.edu>\r
+References: <1340555366-25891-1-git-send-email-sascha-pgp@silbe.org>\r
+       <87pq8n1de4.fsf@awakening.csail.mit.edu>\r
+       <toed34nyr8r.fsf@twin.sascha.silbe.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: 8bit\r
+In-Reply-To: <toed34nyr8r.fsf@twin.sascha.silbe.org>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFlrBKsWRmVeSWpSXmKPExsUixG6nrntG5KW/wew7ehbXb85ktnj77Aaj\r
+       A5PHs1W3mD02/v3BEsAUxWWTkpqTWZZapG+XwJWx4O58poKlyhW9xy+yNTBukOli5OSQEDCR\r
+       2NvxhBnCFpO4cG89WxcjF4eQwD5GieN7G5ggnA2MEnMXnWKGcE4ySRy+cBasRUhgCaPE4h0x\r
+       IDaLgKrE6cezGUFsNgENiW37l4PZIgJmEus3TwKrZwaqaVx7EcwWFnCXWPjmAhOIzSugI9E3\r
+       9R/UtpmMEgfad0ElBCVOznzCAtGsLvFn3iWgZg4gW1pi+T8OiLC8RPPW2WAzOYHeufUdYpeo\r
+       gIrElJPb2CYwCs9CMmkWkkmzECbNQjJpASPLKkbZlNwq3dzEzJzi1GTd4uTEvLzUIl1jvdzM\r
+       Er3UlNJNjKBI4JTk28H49aDSIUYBDkYlHl6P+hf+QqyJZcWVuYcYJTmYlER5Q4Vf+gvxJeWn\r
+       VGYkFmfEF5XmpBYfYpTgYFYS4d19A6icNyWxsiq1KB8mJc3BoiTOeyXlpr+QQHpiSWp2ampB\r
+       ahFMVoaDQ0mCVwgY8UKCRanpqRVpmTklCGkmDk6Q4TxAw5VAaniLCxJzizPTIfKnGHU51r05\r
+       coNRiCUvPy9VSpxXBaRIAKQoozQPbg4sgb1iFAd6S5j3K8gPPMDkBzfpFdASJqAlHJtAPigu\r
+       SURISTUwruDRUuCeeF7qj1f8P1b90NIfXtba66JqfliF7HlyybRTUlnFuGSPvHu5L8dJvpI3\r
+       wkm+F/1nn5C/edic6cCUpocR4cUmmy2/Rp/lZHMLNskzsLOp/OqatOHqK5Utd3+4bjNfvPq+\r
+       wfSL3SoOUTckHfcHxfVZpL83tTy/v/O1zcHWS2f0XuUqsRRnJBpqMRcVJwIANU+9gjsDAAA=\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 26 Jun 2012 01:48:00 -0000\r
+\r
+Quoth Sascha Silbe on Jun 26 at 12:13 am:\r
+> Austin Clements <amdragon@MIT.EDU> writes:\r
+> > On Sun, 24 Jun 2012, Sascha Silbe <sascha-pgp@silbe.org> wrote:\r
+> \r
+> ["notmuch new" listing every directory, even if it's unchanged]\r
+> > I haven't looked over your patches yet, but this result surprises me.\r
+> > Could you explain your setup a little more?  How much mail do you have\r
+> > and across how many directories?  What file system are you using?\r
+> \r
+> As mentioned in passing already, I have a total of about 900k unique\r
+> mails (sometimes several copies of them, received over different paths,\r
+> e.g. mailing list and a direct CC). Most of that is "old" mails, in\r
+> directories that are not getting updated. If notmuch would support mbox,\r
+> I'd use that instead for those old mails. The total number of\r
+> directories in the mail store is about 29k and the total number of files\r
+> (including the git repository and mbox files that sup used) is about\r
+> 1.25M.\r
+> \r
+> Since a housekeeping job last weekend, the number of mails in\r
+> directories that are still getting updated is about 4k, i.e. about 5‰ of\r
+> the total number of mails or 3‰ of the total number of files. The number\r
+> of directories getting updated is 104, i.e. about 4‰ of the total number\r
+> of directories.\r
+> \r
+> Ideally, we'd get the run-time of "notmuch new" down by a similar\r
+> factor. With just plain POSIX and no additional information that won't\r
+> be possible, but providing a way to channel information about updates\r
+> into notmuch (rather than having it scan everything over and over again)\r
+> should help. That information is already available as output from the\r
+> mail fetching process (rsync in my case). Of course, it would be purely\r
+> optional: "notmuch new" without additional information would simply\r
+> continue to scan everything.\r
+\r
+This would be great.  I've been thinking along similar lines for a\r
+while (in my case, I want to feed notmuch new from inotify), though I\r
+haven't written any code for it.\r
+\r
+> > I'm also surprised that your new approach helps.  This directory listing\r
+> > has to be read off disk one way or the other, but listing directories is\r
+> > the bread-and-butter of file systems, whereas I would think that Xapian\r
+> > would require more IO to accomplish the same effect.\r
+> \r
+> "notmuch new" needs to iterate over a list of all directories to find\r
+> those with new mails (and potentially new subdirectories). However, it\r
+> does not need to list the *contents* of those folders. I'm surprised as\r
+> well, but rather in the opposite direction: Based on a naive\r
+> calculation, we'd expect to see a speedup on the order of\r
+> (1.25M+29k)/29k = 44. The actual results suggest that stat()ing (done\r
+> 29k times both before and after the patch) is taking about 19 times as\r
+> long as listing a directory entry (before the patch we listed 1M\r
+> entries, now we list none if nothing has changed). (*)\r
+\r
+For a cold cache, these aren't the numbers that matter.  With an HDD\r
+and how few files your directories contain on average, only seeks will\r
+matter.  I would expect your workload without your patch to have at\r
+least 1 but closer to 2 seeks per directory: one to stat the directory\r
+and one to get the directory contents block.  Some of the stat seeks\r
+will be eliminated by the buffer cache, even starting cold, because of\r
+inode locality (absolute best case is 16x reduction, but if you\r
+created the directories over time, then this locality is probably\r
+quite poor).  There are a few other potential seeks to get the\r
+directory document from Xapian and to get its mtime value, but those\r
+should exhibit strong locality, so they probably don't contribute\r
+much.  NewEgg says your drive has an average seek time of 8.9ms, so\r
+with 29k directories and assuming your directories are sequential on\r
+disk, that's at least 258s and closer to 512s, which agrees with your\r
+benchmark results.\r
+\r
+I'm surprised by your results because I would expect your workload\r
+with your patches to exhibit about the same number of seeks: one to\r
+stat the directory (same as before) and one for\r
+notmuch_directory_get_child_files, which has to seek in the term index\r
+to get the child directories.  My guess is that this exhibits better\r
+locality because the child directory terms are stored contiguously in\r
+the database's key space (though not necessarily sequentially on disk\r
+unless this is a fresh database).\r
+\r
+Unfortunately, I'm not sure of a good way to test this hypothesis.\r
+Any thoughts?\r