From: Pieter Praet <pieter@praet.org>
To: Mueen Nawaz <mueen@nawaz.org>, notmuch@notmuchmail.org
Subject: Re: Questions about importing mail (mbox)
In-Reply-To: <87bp15m9oz.fsf@fester.com>
References: <87bp15m9oz.fsf@fester.com>
User-Agent: Notmuch/0.5-49-g4aeb291 (http://notmuchmail.org) Emacs/23.1.50.1
	(x86_64-pc-linux-gnu)
Date: Mon, 21 Mar 2011 15:31:26 +0100
Message-ID:
 <87zkooo88x.fsf@A7GMS.i-did-not-set--mail-host-address--so-tickle-me>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Precedence: list

On Sun, 20 Mar 2011 20:30:52 -0700, Mueen Nawaz <mueen@nawaz.org> wrote:
> 
> Hi,
> 
> I'm trying to experiment with notmuch. 
> 
> As I understand it, notmuch does not handle mbox for input. The problem
> is that all my mail is currently in mbox format.
> 
> So I first tried converting mbox to maildir using mb2md.
> 
> It didn't do a good job. When I subsequently tried importing to notmuch,
> notmuch complained about lots of non-mail files - I confirmed that
> indeed mb2md had botched converting those emails.
> 
> So then I tried to convert to mh format using Sylpheed. This seemed to
> go well, but then when importing to notmuch, it complained again for
> about 20 emails, and a manual check confirmed that some messages did not
> get converted properly to mh (they don't show up in Sylpheed).
> 
> And then I noticed another discrepancy. mutt shows that I started with
> 44473 messages in mbox. When I imported into Sylpheed, it showed 44482
> messages (no idea where the extra 9 came from). However, notmuch is
> reporting that it processed 44482 files, but that it added 35602
> messages.
> 
> Why only 35602 (it complained for only about 20 messages)? A search
> confirmed that some messages that show up in both mutt (in mbox) and
> Sylpheed (in mh format) were not indexed.
> 
> So I want to know: When you guys switched to notmuch, how did you ensure
> you did not miss any emails. I really, really, really don't want to lose
> any emails in this process!
> 
> Thanks.
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


It would've been a no-brainer if you'd been using Maildir all along
(mbox is evil incarnate), but...

I'd suggest keeping your original mbox file safe in git [1], and
consistently commiting every step of the way, so even if messages were
to get lost in translation, you still have a way to get them back, with
negligible storage overhead (just remember to "git gc --aggressive
--prune=now" when you're finished).

Compacting the mbox file, i.e. purging all stale messages (sync-mailbox
in mutt?) and diffing to HEAD could then possibly give you an indication
as to the origin of the 9 surplus files.

For the actual conversion to Maildir (and any type of mail fetching in
general), I'd suggest using FDM [2], you'll never look back.

Regarding the significant discrepancy between processed and added files
in Notmuch: Could be dupes (e.g. mail to/cc/bcc yourself or mailing
lists, ending up in both Inbox and Sent), which are automatically
suppressed by Notmuch.


[1] http://git-scm.com/
[2] http://fdm.sourceforge.net/