Re: [notmuch] 25 minutes load time with emacs -f notmuch

diff --git a/ca/4564a7d748825fee73cfb52514874e67381691 b/ca/4564a7d748825fee73cfb52514874e67381691
new file mode 100644 (file)
index 0000000..4d57555
--- /dev/null
+++ b/ca/4564a7d748825fee73cfb52514874e67381691
@@ -0,0 +1,86 @@
+Return-Path: <cworth@cworth.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id D20A4431FBF;\r
+       Sat, 21 Nov 2009 19:28:31 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id SefZV3xd2Lqu; Sat, 21 Nov 2009 19:28:31 -0800 (PST)\r
+Received: from cworth.org (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 5D685431FAE;\r
+       Sat, 21 Nov 2009 19:28:30 -0800 (PST)\r
+From: Carl Worth <cworth@cworth.org>\r
+To: Brett Viren <brett.viren@gmail.com>\r
+In-Reply-To: <46263c600911211436s5826015eqc5fc18a4164245cb@mail.gmail.com>\r
+References: <20091121145111.GB19397@excalibur.local>\r
+       <87fx874xj5.fsf@yoom.home.cworth.org>\r
+       <46263c600911211436s5826015eqc5fc18a4164245cb@mail.gmail.com>\r
+Date: Sun, 22 Nov 2009 04:28:18 +0100\r
+Message-ID: <87hbsn2q7h.fsf@yoom.home.cworth.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Cc: notmuch@notmuchmail.org\r
+Subject: Re: [notmuch] 25 minutes load time with emacs -f notmuch\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.12\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 22 Nov 2009 03:28:32 -0000\r
+\r
+On Sat, 21 Nov 2009 17:36:18 -0500, Brett Viren <brett.viren@gmail.com> wrote:\r
+> Processed 130871 total files in 38m 7s (57 files/sec.).\r
+> Added 102723 new messages to the database (not much, really).\r
+\r
+Just be glad that you have so little mail. ;-)\r
+\r
+> This was ~2GB of mail on a 2.5GHz CPU.  That seems pretty reasonable\r
+> to me but I'd like to rerun the "notmuch new" under google perftools\r
+> to see if there are any obvious bottlenecks that might be cleaned up.\r
+\r
+To me, here are the obvious things to fix after looking at a profile:\r
+\r
+  1. We're spending a *lot* of time searching in the Xapian database.\r
+\r
+But our initial indexing operation should only be *writing* data into\r
+the database, so what's this searching about?\r
+\r
+Well, at each new message, we're looking up the ID from it's In-Reply-To\r
+header to find a thread-ID to link to, and then we're looking up all of\r
+the IDs from its References header to find thread IDs that need to be\r
+merged with ours. So both parent and child lookups.\r
+\r
+And since those are taking a bunch of time, I think it might make sense\r
+to just keep a hashtable mapping message-ID -> thread-ID and do lookups\r
+in that, (should have plenty of memory on current machines even with\r
+lots of mail).\r
+\r
+  2. We're hitting the slow Xapian document updates for thread-ID\r
+  merging.\r
+\r
+Whenever we find a child that was already in the database with one\r
+thread ID that should have ours, we simply want to set its thread ID to\r
+ours. But as we've talked about recently, Xapian has a bug (defect 250)\r
+that makes it much more expensive than it should be to update a single\r
+term.\r
+\r
+So, we could do a first pass over the messages to find all their thread\r
+IDs and get them to settle down before doing any indexing in a separate,\r
+second pass.\r
+\r
+Step (2) should help even if we don't do step (1), but clearly we can do\r
+both.\r
+\r
+It would be great if anyone wants to take a look at either or both of\r
+these, otherwise I will when I can.\r
+\r
+-Carl\r