DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type;
	b=TstgJ6bOxTVhyjX5+Rz/Hfu0c46PpBerJvVcOV2OKD2SwmdstWHbyULy1067fXK/FI
	a7hAgdyyCjrBpXkrwoxE9TStzLeVPRz1Tk9PoAP19W+b6Yard5M6LJvJD5PWKFMBWuo1
	DEu62k99FNjOrdt0miNKzFzCmvFGvHWIYpVFI=
MIME-Version: 1.0
In-Reply-To: <87skc8oqyn.fsf@yoom.home.cworth.org>
References: <123554aa0911200056h73def158pb0db64a2a78ed687@mail.gmail.com>
	<87skc8oqyn.fsf@yoom.home.cworth.org>
Date: Mon, 23 Nov 2009 17:26:41 +0100
Message-ID: <123554aa0911230826o11e54d5ckc90e5ae8dab6ffd3@mail.gmail.com>
From: Dominik Epple <dominik.epple@googlemail.com>
To: Carl Worth <cworth@cworth.org>
Content-Type: text/plain; charset=ISO-8859-1
Cc: notmuch@notmuchmail.org
Subject: Re: [notmuch] notmuch new: Memory problem
Precedence: list

Hi,

2009/11/20 Carl Worth <cworth@cworth.org>:
> On Fri, 20 Nov 2009 09:56:50 +0100, Dominik Epple <dominik.epple@googlemail.com> wrote:
>> Is there a problem with the number of my mails? I currently have over
>> 40.000 Mails... they live currently in mbox files, I created a Maildir
>> with mb2md-3.20.pl.
>
> I'm suspecting that you have some big files in there, (such as indexes
> from some other mail program). We had code in notmuch to detect and
> ignore these, but a recent bug had broken that.
>
> I just fixed this code as of the below commit. So please update and try
> again and let us know if things work any better.

Ok, one of the problems seems to be solved. One can learn from the
info: output that the code actually ignores non-email data. These
files are small and fragments of real mail. Obviously the mb2md code
made errors there.

But I run in a different issue. I have a lot of files in the Maildir
which contain base64 encoded binary data. (Some remote site sends my
its daily backup logs.) Those files are all of 2.4 megabyte in size.
By adding some debug code to notmuch-new.c, I find out that the
program becomes very slow and consumes a lot of memory when adding
these files. I just killed it when it consumed 2 GByte again.

So as you suspected, the problem seems to stem from large files. But
those large files are not indices or stuff like that from different
mail programs, but they are valid emails which contain a lot of
(encoded) binary data.

Perhaps we should be able to configure notmuch such that he ignores
all mails that match specific pattern (like "Subject: Backup logs
from.*")

Regards
Dominik