--- /dev/null
+Return-Path: <sascha-ml-email-notmuch-notmuch@silbe.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 0AAB6431FB6\r
+ for <notmuch@notmuchmail.org>; Mon, 25 Jun 2012 15:14:20 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id qfDweLdEfyDB for <notmuch@notmuchmail.org>;\r
+ Mon, 25 Jun 2012 15:14:19 -0700 (PDT)\r
+Received: from smtp.chost.de (setoy.chost.de [217.160.209.225])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id CAFBA431FAF\r
+ for <notmuch@notmuchmail.org>; Mon, 25 Jun 2012 15:14:18 -0700 (PDT)\r
+Received: (qmail 13902 invoked by uid 5015); 25 Jun 2012 22:14:16 -0000\r
+Received: (nullmailer pid 27762 invoked by uid 123);\r
+ Mon, 25 Jun 2012 22:14:16 -0000\r
+Received: from twin.sascha.silbe.org (twin.sascha.silbe.org [192.168.1.2])\r
+ by flatty.sascha.silbe.org ([192.168.1.252])\r
+ with SMTP via TCP; 25 Jun 2012 22:14:16 -0000\r
+Received: (nullmailer pid 3812 invoked by uid 8193);\r
+ Mon, 25 Jun 2012 22:14:16 -0000\r
+To: Austin Clements <amdragon@MIT.EDU>, notmuch <notmuch@notmuchmail.org>\r
+Subject: Re: [PATCH 0/3] Speed up notmuch new for unchanged directories\r
+In-Reply-To: <87pq8n1de4.fsf@awakening.csail.mit.edu>\r
+References: <1340555366-25891-1-git-send-email-sascha-pgp@silbe.org>\r
+ <87pq8n1de4.fsf@awakening.csail.mit.edu>\r
+User-Agent: Notmuch/0.13.2+51~gecf7cfe (http://notmuchmail.org) Emacs/23.2.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Tue, 26 Jun 2012 00:13:40 +0200\r
+Message-ID: <toed34nyr8r.fsf@twin.sascha.silbe.org>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; boundary="=-=-=";\r
+ micalg=pgp-sha512; protocol="application/pgp-signature"\r
+From: Sascha Silbe <sascha-ml-reply-to-2012-3@silbe.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 25 Jun 2012 22:14:20 -0000\r
+\r
+--=-=-=\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+Austin Clements <amdragon@MIT.EDU> writes:\r
+\r
+> On Sun, 24 Jun 2012, Sascha Silbe <sascha-pgp@silbe.org> wrote:\r
+\r
+["notmuch new" listing every directory, even if it's unchanged]\r
+> I haven't looked over your patches yet, but this result surprises me.\r
+> Could you explain your setup a little more? How much mail do you have\r
+> and across how many directories? What file system are you using?\r
+\r
+As mentioned in passing already, I have a total of about 900k unique\r
+mails (sometimes several copies of them, received over different paths,\r
+e.g. mailing list and a direct CC). Most of that is "old" mails, in\r
+directories that are not getting updated. If notmuch would support mbox,\r
+I'd use that instead for those old mails. The total number of\r
+directories in the mail store is about 29k and the total number of files\r
+(including the git repository and mbox files that sup used) is about\r
+1.25M.\r
+\r
+Since a housekeeping job last weekend, the number of mails in\r
+directories that are still getting updated is about 4k, i.e. about 5=E2=80=\r
+=B0 of\r
+the total number of mails or 3=E2=80=B0 of the total number of files. The n=\r
+umber\r
+of directories getting updated is 104, i.e. about 4=E2=80=B0 of the total n=\r
+umber\r
+of directories.\r
+\r
+Ideally, we'd get the run-time of "notmuch new" down by a similar\r
+factor. With just plain POSIX and no additional information that won't\r
+be possible, but providing a way to channel information about updates\r
+into notmuch (rather than having it scan everything over and over again)\r
+should help. That information is already available as output from the\r
+mail fetching process (rsync in my case). Of course, it would be purely\r
+optional: "notmuch new" without additional information would simply\r
+continue to scan everything.\r
+\r
+\r
+> I'm also surprised that your new approach helps. This directory listing\r
+> has to be read off disk one way or the other, but listing directories is\r
+> the bread-and-butter of file systems, whereas I would think that Xapian\r
+> would require more IO to accomplish the same effect.\r
+\r
+"notmuch new" needs to iterate over a list of all directories to find\r
+those with new mails (and potentially new subdirectories). However, it\r
+does not need to list the *contents* of those folders. I'm surprised as\r
+well, but rather in the opposite direction: Based on a naive\r
+calculation, we'd expect to see a speedup on the order of\r
+(1.25M+29k)/29k=C2=A0=3D=C2=A044. The actual results suggest that stat()ing=\r
+ (done\r
+29k times both before and after the patch) is taking about 19 times as\r
+long as listing a directory entry (before the patch we listed 1M\r
+entries, now we list none if nothing has changed). (*)\r
+\r
+In practice, the speedup achieved by my patch is larger than what the\r
+benchmark suggests because there are other processes running that use\r
+RAM. If we need to read a lot from disk (like "notmuch new" did before\r
+my patch), there's a good chance it's already been evicted from the\r
+cache since the last run. The fewer we need to read, the more likely it\r
+is to still be in the cache. Similarly, reading lots of data from disk\r
+will displace other data in the cache. These effects are not covered by\r
+the pure "hot cache" and "cold cache" timings.\r
+\r
+\r
+> Does your patch win because you can specifically list subdirectories\r
+> out of Xapian, making the IO proportional to the number of\r
+> subdirectories instead of the number of subdirectories and files (even\r
+> though the constant factors probably favor reading from the file\r
+> system)?\r
+\r
+It wins because the factor is the number of files in each directory, not\r
+just some low constant based on file system overhead vs. Xapian\r
+overhead.\r
+\r
+\r
+> I like the idea of these patches, I just want to make sure I have a firm\r
+> grip on what's being optimized and why it wins.\r
+\r
+Certainly a good idea. Thanks for taking the time!\r
+\r
+Sascha\r
+\r
+(*) float(linsolve([29000*x + 1250000*y =3D 3.3 * 29000*x], [x])); in\r
+ maxima, if you'd like to check the math.\r
+=2D-=20\r
+http://sascha.silbe.org/\r
+http://www.infra-silbe.de/\r
+\r
+--=-=-=\r
+Content-Type: application/pgp-signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.10 (GNU/Linux)\r
+\r
+iQEcBAEBCgAGBQJP6OKVAAoJELpz82VMF3DaMzIIALpqmnz26Mk8EZMooszj6oOK\r
+rA6b+LO+B9qCdpSc1/0bs/qm7pC1AEs3G6ycliqntUddj34vq0jXW+yZ2llou6kk\r
+W56B4fVnamYX+AtFSrNHi9GxRcyDRK6fmZv5Qtr55poJayKFaeJNhaj4EblULtCp\r
+3JeEQI+x9FJglVMMp67QTZMlrn0JIxqyfeWDhbpBYdunJrraOtF3hmJeqfJbIcMm\r
+5rDkvwcvybjjP1oA5wHN/H8euoFb0CO0K+Y36MCiemu0xnijlGaUVt6/I/wjNn1F\r
+yesV4CQHZ5VsBKWYeLxV3BRETUDKvN5ds/gjffbZhoiJSShA/hCbYHPhj7jjdUc=\r
+=8WVY\r
+-----END PGP SIGNATURE-----\r
+--=-=-=--\r
projects / notmuch-archives.git / commitdiff