DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com;
	h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
	b=MEojAI/vl5NuBVAoB+5TD2gTNni+HStoIfGzUACU48Kfpwvjnb1d/aFK7kzfZkl22Re9uM9tm5/P008fObQ9KSo8Q++ComnAs1z6MQijSvh5OlNF7+VJLhFo1qFgT8LjwzcdVhh2rzOf90rBz5L9J+QPhTcNs4fIBHnzSa/UbAY=;
References: <1321930927.73603.YahooMailNeo@web36506.mail.mud.yahoo.com>
	<CAMP44s2zf-OpN4u+zv1g+q_nrgkMHUGLUT7cVFZMJ=Xdy-9WdA@mail.gmail.com>
Message-ID: <1322068833.15983.YahooMailNeo@web36504.mail.mud.yahoo.com>
Date: Wed, 23 Nov 2011 09:20:33 -0800 (PST)
From: Tom Bulli <mrbulli@yahoo.com>
Subject: Re: Notmuch indexing 21 million emails
To: Felipe Contreras <felipe.contreras@gmail.com>
In-Reply-To:
 <CAMP44s2zf-OpN4u+zv1g+q_nrgkMHUGLUT7cVFZMJ=Xdy-9WdA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: "notmuch@notmuchmail.org" <notmuch@notmuchmail.org>
Precedence: list
Reply-To: Tom Bulli <mrbulli@yahoo.com>

I have been able to speed that up with the code below - basically increase =
"XAPIAN_FLUSH_THRESHOLD" based on the total virtual memory divided by the a=
vg. size of an email times 2 (just to be safe).=A0 It seems to be faster si=
nce it does less xapian updates.=A0 However, I have a nagging feeling that =
""XAPIAN_FLUSH_THRESHOLD" could even be higher since I don't see any increa=
se in used memory (via "top -d 1").=A0 The server in question has eight CPU=
 cores and 8GB RAM, running Debian squeeze on a 32bit architecture (I know =
- but it is what it is :) ).=0A=0A=0A# Assume an average size of 120KB per =
email=0A#=A0 and use at most half the virtual memory=0AXFT=3D$(($(free -otk=
=A0 | awk '/^Total/ {print $2}') / 240))=0A# Keep more index info in memory=
 before flushing to disk=0A[ $XFT -lt 10000 ] && XFT=3D10000=0Asu - archive=
 -c "export XAPIAN_FLUSH_THRESHOLD=3D$XFT; notmuch new --verbose"=0A=0A=0A=
=0A=0A----- Original Message -----=0A> From: Felipe Contreras <felipe.contr=
eras@gmail.com>=0A> To: Tom Bulli <mrbulli@yahoo.com>=0A> Cc: "notmuch@notm=
uchmail.org" <notmuch@notmuchmail.org>=0A> Sent: Wednesday, November 23, 20=
11 10:40 AM=0A> Subject: Re: Notmuch indexing 21 million emails=0A> =0A> On=
 Tue, Nov 22, 2011 at 5:02 AM, Tom Bulli <mrbulli@yahoo.com> wrote:=0A>>  I=
 have a project where I need to search about 21 emails - and decided to =0A=
> use "notmuch" for it.=A0 The system is a Debian Squeeze, the notmuch =0A>=
 version is "0.8-1~bpo60+1" from "kyria's" private =0A> repository.=0A>> =
=0A>>  I am running the "notmuch new" for approx. 4 days now - and =0A> acc=
ording to "not,uch count" it has indexed about 4.5 million emails.=0A>> =0A=
>>  Is this expected performance?=A0 Is there any way to speed that up?=0A>=
 =0A> It would be nice to run something like this with OProfile (or perf)=
=0A> and see if there's some obvious fixes.=0A> =0A> -- =0A> Felipe Contrer=
as=0A>