1 Return-Path: <cworth@cworth.org>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by olra.theworths.org (Postfix) with ESMTP id D20A4431FBF;
\r
6 Sat, 21 Nov 2009 19:28:31 -0800 (PST)
\r
7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
\r
8 Received: from olra.theworths.org ([127.0.0.1])
\r
9 by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
\r
10 with ESMTP id SefZV3xd2Lqu; Sat, 21 Nov 2009 19:28:31 -0800 (PST)
\r
11 Received: from cworth.org (localhost [127.0.0.1])
\r
12 by olra.theworths.org (Postfix) with ESMTP id 5D685431FAE;
\r
13 Sat, 21 Nov 2009 19:28:30 -0800 (PST)
\r
14 From: Carl Worth <cworth@cworth.org>
\r
15 To: Brett Viren <brett.viren@gmail.com>
\r
16 In-Reply-To: <46263c600911211436s5826015eqc5fc18a4164245cb@mail.gmail.com>
\r
17 References: <20091121145111.GB19397@excalibur.local>
\r
18 <87fx874xj5.fsf@yoom.home.cworth.org>
\r
19 <46263c600911211436s5826015eqc5fc18a4164245cb@mail.gmail.com>
\r
20 Date: Sun, 22 Nov 2009 04:28:18 +0100
\r
21 Message-ID: <87hbsn2q7h.fsf@yoom.home.cworth.org>
\r
23 Content-Type: text/plain; charset=us-ascii
\r
24 Cc: notmuch@notmuchmail.org
\r
25 Subject: Re: [notmuch] 25 minutes load time with emacs -f notmuch
\r
26 X-BeenThere: notmuch@notmuchmail.org
\r
27 X-Mailman-Version: 2.1.12
\r
29 List-Id: "Use and development of the notmuch mail system."
\r
30 <notmuch.notmuchmail.org>
\r
31 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
\r
32 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
33 List-Archive: <http://notmuchmail.org/pipermail/notmuch>
\r
34 List-Post: <mailto:notmuch@notmuchmail.org>
\r
35 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
36 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
\r
37 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
38 X-List-Received-Date: Sun, 22 Nov 2009 03:28:32 -0000
\r
40 On Sat, 21 Nov 2009 17:36:18 -0500, Brett Viren <brett.viren@gmail.com> wrote:
\r
41 > Processed 130871 total files in 38m 7s (57 files/sec.).
\r
42 > Added 102723 new messages to the database (not much, really).
\r
44 Just be glad that you have so little mail. ;-)
\r
46 > This was ~2GB of mail on a 2.5GHz CPU. That seems pretty reasonable
\r
47 > to me but I'd like to rerun the "notmuch new" under google perftools
\r
48 > to see if there are any obvious bottlenecks that might be cleaned up.
\r
50 To me, here are the obvious things to fix after looking at a profile:
\r
52 1. We're spending a *lot* of time searching in the Xapian database.
\r
54 But our initial indexing operation should only be *writing* data into
\r
55 the database, so what's this searching about?
\r
57 Well, at each new message, we're looking up the ID from it's In-Reply-To
\r
58 header to find a thread-ID to link to, and then we're looking up all of
\r
59 the IDs from its References header to find thread IDs that need to be
\r
60 merged with ours. So both parent and child lookups.
\r
62 And since those are taking a bunch of time, I think it might make sense
\r
63 to just keep a hashtable mapping message-ID -> thread-ID and do lookups
\r
64 in that, (should have plenty of memory on current machines even with
\r
67 2. We're hitting the slow Xapian document updates for thread-ID
\r
70 Whenever we find a child that was already in the database with one
\r
71 thread ID that should have ours, we simply want to set its thread ID to
\r
72 ours. But as we've talked about recently, Xapian has a bug (defect 250)
\r
73 that makes it much more expensive than it should be to update a single
\r
76 So, we could do a first pass over the messages to find all their thread
\r
77 IDs and get them to settle down before doing any indexing in a separate,
\r
80 Step (2) should help even if we don't do step (1), but clearly we can do
\r
83 It would be great if anyone wants to take a look at either or both of
\r
84 these, otherwise I will when I can.
\r
projects / notmuch-archives.git / blob