--- /dev/null
+Return-Path: <sascha-ml-email-notmuch-notmuch@silbe.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 2D515431FB6\r
+ for <notmuch@notmuchmail.org>; Thu, 5 Jul 2012 00:37:54 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id UhgVPmhv0SUU for <notmuch@notmuchmail.org>;\r
+ Thu, 5 Jul 2012 00:37:53 -0700 (PDT)\r
+Received: from smtp.chost.de (setoy.chost.de [217.160.209.225])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id DD975431FAE\r
+ for <notmuch@notmuchmail.org>; Thu, 5 Jul 2012 00:37:52 -0700 (PDT)\r
+Received: (qmail 1719 invoked by uid 5015); 5 Jul 2012 07:37:48 -0000\r
+Received: (nullmailer pid 19070 invoked by uid 123);\r
+ Thu, 05 Jul 2012 07:37:48 -0000\r
+Received: from twin.sascha.silbe.org (twin.sascha.silbe.org [192.168.1.2])\r
+ by flatty.sascha.silbe.org ([192.168.1.252])\r
+ with SMTP via TCP; 05 Jul 2012 07:37:48 -0000\r
+Received: (nullmailer pid 10893 invoked by uid 8193);\r
+ Thu, 05 Jul 2012 07:37:48 -0000\r
+To: Austin Clements <amdragon@MIT.EDU>, notmuch <notmuch@notmuchmail.org>\r
+Subject: Re: [PATCH 0/3] Speed up notmuch new for unchanged directories\r
+In-Reply-To: <20120626014754.GO24342@mit.edu>\r
+References: <1340555366-25891-1-git-send-email-sascha-pgp@silbe.org>\r
+ <87pq8n1de4.fsf@awakening.csail.mit.edu>\r
+ <toed34nyr8r.fsf@twin.sascha.silbe.org>\r
+ <20120626014754.GO24342@mit.edu>\r
+User-Agent: Notmuch/0.13.2+51~gecf7cfe (http://notmuchmail.org) Emacs/23.2.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Thu, 05 Jul 2012 09:37:22 +0200\r
+Message-ID: <toemx3evetp.fsf@twin.sascha.silbe.org>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; boundary="=-=-=";\r
+ micalg=pgp-sha512; protocol="application/pgp-signature"\r
+From: Sascha Silbe <sascha-ml-reply-to-2012-3@silbe.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 05 Jul 2012 07:37:54 -0000\r
+\r
+--=-=-=\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+Austin Clements <amdragon@MIT.EDU> writes:\r
+\r
+>> "notmuch new" needs to iterate over a list of all directories to find\r
+>> those with new mails (and potentially new subdirectories). However, it\r
+>> does not need to list the *contents* of those folders. I'm surprised as\r
+>> well, but rather in the opposite direction: Based on a naive\r
+>> calculation, we'd expect to see a speedup on the order of\r
+>> (1.25M+29k)/29k=C2=A0=3D=C2=A044. The actual results suggest that stat()=\r
+ing (done\r
+>> 29k times both before and after the patch) is taking about 19 times as\r
+>> long as listing a directory entry (before the patch we listed 1M\r
+>> entries, now we list none if nothing has changed). (*)\r
+>\r
+> For a cold cache, these aren't the numbers that matter. With an HDD\r
+> and how few files your directories contain on average, only seeks will\r
+> matter. I would expect your workload without your patch to have at\r
+> least 1 but closer to 2 seeks per directory: one to stat the directory\r
+> and one to get the directory contents block.\r
+[...]\r
+> I'm surprised by your results because I would expect your workload\r
+> with your patches to exhibit about the same number of seeks: one to\r
+> stat the directory (same as before) and one for\r
+> notmuch_directory_get_child_files, which has to seek in the term index\r
+> to get the child directories. My guess is that this exhibits better\r
+> locality because the child directory terms are stored contiguously in\r
+> the database's key space (though not necessarily sequentially on disk\r
+> unless this is a fresh database).\r
+\r
+Well, what I'd expect from the file system is good locality for large\r
+files. So once you are at the database index, reading is fast (about\r
+50MB/s). But the directories are dispersed and were created over time,\r
+so their overall locality (remember that we need only a few blocks per\r
+directory) is likely to be rather poor.\r
+\r
+I would have to take a closer look at ext4 (the last time I looked at\r
+the on-disk structures of ext* was several years ago), but IIRC\r
+everything we need for the stat() should be either in its inode. So the\r
+patch saves a read of the leaf directory data blocks. I'd expect leaf\r
+directories to outnumber intermediate directories by about 3-4:1 (cur/,\r
+new/, tmp/ and additionally new-20120623 in my case). That would be\r
+consistent with the speedup, but may just be coincidence.\r
+\r
+What we'd really need to know here (to understand the particular numbers\r
+my tests resulted in) is the block allocation strategies of ext4.\r
+\r
+\r
+> Unfortunately, I'm not sure of a good way to test this hypothesis.\r
+> Any thoughts?\r
+\r
+The only scientific way to test it that I can think of would be to\r
+determine the actual block numbers on disk (does FIEMAP work for\r
+directories?) to calculate locality and / or record access times during\r
+the "notmuch new" run. However that would gather a rather large amount\r
+of data, requiring a custom tool to condense it into something we (as\r
+humans) can understand. I don't have time to write the analysis tool,\r
+but if someone else does I'm happy to provide the data.\r
+\r
+An engineer would probably do a test series on various hardware\r
+instead. Given the wide variety of hardware and file systems that\r
+notmuch should work well with, that would be a good idea in itself.\r
+\r
+Sascha\r
+=2D-=20\r
+http://sascha.silbe.org/\r
+http://www.infra-silbe.de/\r
+\r
+--=-=-=\r
+Content-Type: application/pgp-signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.10 (GNU/Linux)\r
+\r
+iQEcBAEBCgAGBQJP9UQzAAoJELpz82VMF3Dae+cH/jGO/XftuSdBnegEdKOu4P7O\r
+ANKgrfMIBaF8sbX+ftP4LisP6KO8jB/8/mwk4f9vmvvoP1+IMpHcJ2MLE/TzcDlq\r
+H8se7xL1rr1uWOfheVZdB9mBzf0Ag03xcjSxBo9yIbaVMwF6XXeeSXGirhichzhk\r
+6/2Gm9JjoFmjTrm3sQEs/nlV5SlxgyxblzrJWgQIuoDRTl0dw9duDk5zSY28ih3g\r
+u8Y5NwIyBVF9zPMFXMCIyfV2Zwhowv5bt1Vfc0iw+l7mKS0P0YKnShEVAXP48kE9\r
+vd/6cKljSTyWGsQZri1N4jCIL1Kskx3Gfi3376zNaux+CC+7kX6g/D0guFAj1eM=\r
+=ncvA\r
+-----END PGP SIGNATURE-----\r
+--=-=-=--\r
projects / notmuch-archives.git / commitdiff