1 Return-Path: <cworth@cworth.org>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by olra.theworths.org (Postfix) with ESMTP id 37FD7431FBC
\r
6 for <notmuch@notmuchmail.org>; Sun, 10 Jan 2010 09:43:40 -0800 (PST)
\r
7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
\r
8 Received: from olra.theworths.org ([127.0.0.1])
\r
9 by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
\r
10 with ESMTP id 4Dmbhh1dmmsh for <notmuch@notmuchmail.org>;
\r
11 Sun, 10 Jan 2010 09:43:39 -0800 (PST)
\r
12 Received: from yoom.home.cworth.org (localhost [127.0.0.1])
\r
13 by olra.theworths.org (Postfix) with ESMTP id 541D3431FAE
\r
14 for <notmuch@notmuchmail.org>; Sun, 10 Jan 2010 09:43:39 -0800 (PST)
\r
15 Received: by yoom.home.cworth.org (Postfix, from userid 1000)
\r
16 id 0BFCB25400A; Sun, 10 Jan 2010 09:43:39 -0800 (PST)
\r
17 From: Carl Worth <cworth@cworth.org>
\r
18 To: notmuch@notmuchmail.org
\r
19 Date: Sun, 10 Jan 2010 09:43:38 -0800
\r
20 Message-ID: <87ocl1lut1.fsf@yoom.home.cworth.org>
\r
22 Content-Type: multipart/signed; boundary="=-=-=";
\r
23 micalg=pgp-sha1; protocol="application/pgp-signature"
\r
24 Subject: [notmuch] Some Xapian tips and thoughts on rebuilding
\r
25 X-BeenThere: notmuch@notmuchmail.org
\r
26 X-Mailman-Version: 2.1.13
\r
28 List-Id: "Use and development of the notmuch mail system."
\r
29 <notmuch.notmuchmail.org>
\r
30 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
\r
31 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
32 List-Archive: <http://notmuchmail.org/pipermail/notmuch>
\r
33 List-Post: <mailto:notmuch@notmuchmail.org>
\r
34 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
35 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
\r
36 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
37 X-List-Received-Date: Sun, 10 Jan 2010 17:43:40 -0000
\r
40 Content-Transfer-Encoding: quoted-printable
\r
42 With the recent change to "database format 1" some users might decide to
\r
43 rebuild their notmuch database. If so, there are some things I've
\r
44 learned about Xapian that are good to know before you rebuild. Or maybe
\r
45 what you read below will encourage you to rebuild your notmuch database.
\r
47 I think all users of notmuch have been discouraged by how slow it is to
\r
48 change the tags on messages. Many of you have heard of "Xapian defect
\r
49 #250" that was causing some slowness here. I'm happy to report that with
\r
50 initial code from Kan-Ru Chen, Richard Boulton has recently committed a
\r
51 fix for this bug to Xapian upstream, (after rewriting the fix
\r
52 substantially, extending the fix to multiple backends, and writing
\r
53 several new Xapian test cases for it).
\r
55 However, just upgrading your Xapian library won't necessarily give you
\r
56 any benefit with notmuch. But you can be assured of getting some benefit
\r
57 if you upgrade both Xapian and notmuch and rebuild your notmuch
\r
58 database. The gory details are covered below.
\r
60 Gory details for getting the Xapian #250 fix benefit with flint
\r
61 =2D--------------------------------------------------------------
\r
62 Xapian has a notion of multiple backends which store the data in the
\r
63 database differently. In the 1.0 versions of Xapian, the default backend
\r
64 is the "flint" backend. This backend stores the document "length" in
\r
65 every "posting" entry, (where a posting is effectively a link from a
\r
66 particular "term" to a particular "document" perhaps with positional
\r
69 The fix for defect #250 is to update as little as possible when we add
\r
70 or remove a single term (and hence a posting) to a document. But if this
\r
71 change also changes the document length, then all postings will
\r
72 unavoidably need to be updated.
\r
74 Historically, notmuch hasn't taken any special care with the results on
\r
75 "document length" when adding terms for things like tags. The default
\r
76 treatment is that terms *do* affect document length. But for terms like
\r
77 tags that don't actually occur in the document content, it makes sense
\r
78 to record them as having 0 effect on the document length. I recently
\r
79 fixed notmuch to do so. But you'll have to rebuild your notmuch database
\r
80 with a recent notmuch in order to get that change.
\r
82 But if you rebuild, you might want to use chert instead of flint
\r
83 =2D---------------------------------------------------------------
\r
84 I mentioned that "flint" is the default backend in the 1.0 releases of
\r
85 Xapian. In the development versions that you can checkout from the
\r
86 project's svn repository, there's support for a newer backend named
\r
87 "chert", (expected to be the default in an upcoming release). To get
\r
88 Xapian to use chert you need to have the following environment variable
\r
89 set when doing the initial "notmuch new" to build your database:
\r
91 XAPIAN_PREFER_CHERT=3D1
\r
93 After that, Xapian will see that your database is chert and will know
\r
94 how to deal with it. (Except that I have seen that upgrading Xapian
\r
95 From=20one svn version to another may result in incompatible changes to
\r
96 the chert format---so a future Xapian may not be able to read a
\r
97 previously-created chert database. I assume these format changes won't
\r
98 happen in stable releases of Xapian.)
\r
100 One thing that's nice about chert compared to flint is that it no longer
\r
101 stores the document length in every posting. This means it's easier to
\r
102 get the benefit from the Xapian defect #250 fix. It also means that your
\r
103 database can be much smaller. For my notmuch database, a flint built is
\r
104 about 7.0GB while a chert build is only 5.0GB---a very nice change.
\r
106 Compacting your database
\r
107 =2D-----------------------
\r
108 One final tip. I recently started experimenting with a Xapian feature
\r
109 for compacting a database. This is available only via a command-line
\r
110 program, (named xapian-compact in the 1.0 releases and
\r
111 xapian-compact-1.1 in the current Xapian from svn). This functionality
\r
112 is not yet available in the Xapian library interface or else I would
\r
113 probably make notmuch call it after building the database.
\r
115 If you want to experiment with xapian-compact, you'll want to call it
\r
116 with a command something like the following:
\r
118 xapian-compact-1.1 --no-renumber ~/mail/.notmuch/xapian ~/mail/.notmuc=
\r
121 The --no-renumber argument is essential with a notmuch database, since
\r
122 (as of database format version 1), notmuch stores Xapian document IDs
\r
123 internally within terms. If you forget this, you'll find that all of
\r
124 your searches will return results that are unable to locate any of the
\r
125 filenames corresponding to your mail.
\r
127 After running the above command, you could then move your existing
\r
128 .notmuch/xapian away and move .notmuch/xapian-compact in its place to
\r
129 test, and then discard the original .notmuch/xapian if you're happy with
\r
132 For me, this compaction took my 5.0GB down to 3.1GB. So my database is
\r
133 now less than half the size of what I started with with flint, (and can
\r
134 conceivable be cached entirely within memory on my machine!), which is
\r
137 I hope the above is helpful, (and yes, clearly we need to get this
\r
138 content out in other ways such as in a README in the source
\r
139 distribution, and on the website in some form much better than our
\r
140 current pipermail-based mailing-list archives).
\r
145 Content-Type: application/pgp-signature
\r
147 -----BEGIN PGP SIGNATURE-----
\r
148 Version: GnuPG v1.4.10 (GNU/Linux)
\r
150 iD8DBQFLShHK6JDdNq8qSWgRAnnQAJ4ubKP5ePBKqhFnHoyAYw8IlVxG9wCfaBee
\r
151 rRAqf58rk/uD80uGu3Fyg3w=
\r
153 -----END PGP SIGNATURE-----
\r
projects / notmuch-archives.git / blob