From: Carl Worth <cworth@cworth.org>
To: Jeffrey Ollie <jeff@ocjtech.us>, Not Much Mail
	<notmuch@notmuchmail.org>
In-Reply-To: <935ead450911271709p3924d929n15e743c26704ecbe@mail.gmail.com>
References: <935ead450911271709p3924d929n15e743c26704ecbe@mail.gmail.com>
Date: Fri, 27 Nov 2009 22:09:03 -0800
Message-ID: <873a3ztc3k.fsf@yoom.home.cworth.org>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"
Subject: Re: [notmuch] Problems importing my mail...
Precedence: list

--=-=-=

On Fri, 27 Nov 2009 19:09:56 -0600, Jeffrey Ollie <jeff@ocjtech.us> wrote:
> I'm having some problems importing my mail.  I've got quite a bit
> stored up, and some of them I'm sure are quite large.  After several
> hours I get the following.  Is notmuch running out of memory on me?

Comments below (as far as I can make them):

> $ ./notmuch new
> Found 328184 total files.

That's certainly not the largest number of messages we've seen indexed
successfully by notmuch, (I think Keith has near 3 times that
number). [Maybe notmuch should be reporting the total size of the mail
store as well...]

> Warning: Unexpected extra parts of multipart/signed. Indexing anyway.

Oh, that's a warning I put in place because I wasn't sure if it was
legitimate for a multipart/signed message to have more than two
parts. I'd actually be interested to know if the mail is correct, (and I
should just eliminate the warning), or if the mail is somehow malformed
and the warning is correct.

> Note: Ignoring non-mail file:
> /home/jeff/mail/message/6/5/65c74c15a686187bb6bbf9958f494fc6b80068034a659a9ad44991b08c58f2d2
> Note: Ignoring non-mail file:
> /home/jeff/mail/message/7/9/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451
> Note: Ignoring non-mail file:
> /home/jeff/mail/message/8/0/802071f7fcd8b0b74a19e1ca64e5468184fee0c9171bacb77ae1fe1669c426ee

Those you should check to see if they actually do look like mail
messages. Notmuch decides to ignore a file when it can't find any of the
following headers: Subject:, From:, not To:.

> A Xapian exception occurred creating message: Db block overwritten -
> are there multiple writers?
> Error: A Xapian exception occurred. Halting processing.

That's an error I've never seen before. We might want to talk to the
Xapian folks to see what that could be. There's really no way there can
be multiple writers here. So I don't know what the actual problem might
be.

> Internal error: Message with document ID of 175013 has no thread ID.
>  (lib/message.cc:353).
> [jeff@max1 notmuch]$ ./notmuch new
> Internal error: Message with document ID of 175013 has no thread ID.
>  (lib/message.cc:353).

Hmm... we could probably do better here. The fatal error you're getting
here is for an invariant that notmuch thinks is fairly important, (no
mail document should exist without a thread ID). Meanwhile, however when
adding a new message we do actually create a mail document in the
database, and only later resolve its thread ID and add that to the
database as well. A better solution would be to resolve the thread ID
before adding anything to the database so that this invariant would
never be violated.

Some people have been proposing a "notmuch gc" command or so for
cleaning up problems like this.

In the meantime, you could explore the current state of your database by
changing the code that's currently giving you an internal error to
instead return a fake thread ID. For example:

    if (i == message->doc.termlist_end () || id[0] != *prefix)
        message->thread_id = talloc_strdup (message, "00000000000000000000000000000000");
    else
        message->thread_id = talloc_strdup (message, id.c_str () + 1);

Good luck!

-Carl

--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iD8DBQFLEL5/6JDdNq8qSWgRAuCyAJ9asstZ6641C1c8n/Wog47/EaqNYwCeJZ0L
wyBVz4JEZfQ/cDYp5MYTVQ8=
=TlAq
-----END PGP SIGNATURE-----
--=-=-=--