From: Mark Walters <markwalters1009@gmail.com>
To: Eric Gillespie <epg@pretzelnet.org>, notmuch@notmuchmail.org
Subject: Re: nmh + notmuch: expected to work?
In-Reply-To: <24056.1402437857@wundagore>
References: <24056.1402437857@wundagore>
User-Agent: Notmuch/0.15.2+615~g78e3a93 (http://notmuchmail.org) Emacs/23.4.1
	(x86_64-pc-linux-gnu)
Date: Wed, 11 Jun 2014 08:45:25 +0100
Message-ID: <87k38nx52i.fsf@qmul.ac.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Precedence: list


Hello

On Tue, 10 Jun 2014, Eric Gillespie <epg@pretzelnet.org> wrote:
> I'm a long-time nmh user, and I heard notmuch was supposed to
> handle mh folders, so I gave it a shot.
>
> It's crazy slow.  The first 'notmuch new' took 4 hours, but maybe
> that's expected.  However, every 'notmuch new' takes 4 - 10
> hours.  The only time it completes faster is if I immediately
> re-run it without incorporating new mail.  In that case, it
> completes in 0.8s - 38s, which is quite reasonable for my mail.
>
> I have 18.5 GB, 1,058,370 messages in 315 directories.
>
> From the messages it prints, apparently any directory that
> changes (receives a new message) has to have all its files
> indexed from scratch.  What?!

This is definitely not expected behaviour and doesn't happen under
maildir. I would not expect it to happen under mh format but I am not
familiar with that format.

> # re-index after some new mail (< 1000; not 704083!)
> Processed 704083 total files in 11h 7m 48s (17 files/sec.).
> Added 701791 new messages to the database. Removed 701332 messages. Detected 2136 file renames.
>
> # re-index again, no new mail
> Processed 60 total files in 40s (1 files/sec.).
> No new mail.
>
> # incorporate 1 new message
> Processed 9181 total files in 7m 54s (19 files/sec.).
> Added 9071 new messages to the database. Removed 9071 messages. Detected 12 file renames.
>
> # incorporate 1 new message
> Processed 111 total files in almost no time.
> Added 1 new message to the database. Detected 12 file renames.
>
> # incorporate 1 new message
> Processed 757 total files in 44s (17 files/sec.).
> Added 645 new messages to the database. Removed 645 messages. Detected 14 file renames.

Is anything else accessing modifying or doing anything to the MH
hierarchy? Some things (eg http://wiki2.dovecot.org/MailboxFormat/MH)
indicate that some tools do manipulate the hierarchy (eg sortm renames
things). 

if you run notmuch new as 

notmuch new --verbose --debug

it might spit out some more useful information as to what notmuch is
doing (*do* check that nothing it says is sensitive before posting
though!)

Another thing you could try is 
notmuch search --output=messages folder:somefolder 

before and after adding a message to somefolder (*) and seeing if that
shows significant changes. (Note message ids do say a lot about who your
correspondents are so you probably don't want to post the outputs to the
list.)

Best wishes

Mark

(*) the syntax for folder based searches changed recently so it may
depend which version of notmuch you are running: i didn't see it
mentioned in your email.