Date: Tue, 22 Nov 2011 21:48:18 -0500
From: Austin Clements <amdragon@MIT.EDU>
To: Petter Reinholdtsen <pere@hungry.com>
Subject: Re: 'notmuch new' leaking memory and getting slower over time?
Message-ID: <20111123024818.GI9351@mit.edu>
References: <2flfwhht87d.fsf@diskless.uio.no>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2flfwhht87d.fsf@diskless.uio.no>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: notmuch@notmuchmail.org
Precedence: list

Quoth Petter Reinholdtsen on Nov 21 at 11:35 pm:
> The indexing took 36 hours.  At the start it claimed it would take 10
> hours, and it continued to underestimate the amount of time left until
> the very end.  It claimed to have 1 hour left when I checked before I
> went to bed, and claimed to have 15 minutes left when I woke up 6-7
> hours later.

notmuch new does a simple linear extrapolation based on how many files
it's examined and how many there are total.  This is doomed to
undershoot at least because indexing becomes slower as the database
grows (B-tree insertion is O(log N), fragmentation will increase over
time, posting lists will get longer...).

I'm not sure much can be done about the estimate at the beginning,
short of throwing in some fudge factor, but the estimates later in the
process would be much more accurate if it used a sliding window,
rather than measuring from the beginning.

> Shortly before the indexing finished, the notmuch process was using 1.2
> GiB of resident memory according to top.  Is the process leaking memory?

It's possible this is just memory fragmentation, but it definitely
sounds like a leak.  talloc has some tools for tracking down leaks and
it would be good to heap profile notmuch new, but to my knowledge
nobody's applied these tools.