From: Patrick Totzke Date: Tue, 11 Dec 2012 18:26:38 +0000 (+0000) Subject: Re: notmuch python bindings corrupt db index (was: gmail importer script) X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=b8b5efe78186dce67cf822fbd92739d06cc8b399;p=notmuch-archives.git Re: notmuch python bindings corrupt db index (was: gmail importer script) --- diff --git a/48/8c9203c3b99d6d915af2310cb3f091159ca988 b/48/8c9203c3b99d6d915af2310cb3f091159ca988 new file mode 100644 index 000000000..49f659c3c --- /dev/null +++ b/48/8c9203c3b99d6d915af2310cb3f091159ca988 @@ -0,0 +1,137 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 25E7E431FBD + for ; Tue, 11 Dec 2012 10:26:47 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.799 +X-Spam-Level: +X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id BD7mH5h8VfnV for ; + Tue, 11 Dec 2012 10:26:46 -0800 (PST) +Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com + [209.85.212.175]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 51B15431FBC + for ; Tue, 11 Dec 2012 10:26:46 -0800 (PST) +Received: by mail-wi0-f175.google.com with SMTP id hm11so2094754wib.2 + for ; Tue, 11 Dec 2012 10:26:43 -0800 (PST) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; + h=content-type:mime-version:content-disposition:to:from:in-reply-to + :cc:references:message-id:user-agent:subject:date; + bh=R9pZyJkU0CFmldU2joZA0jvKor0C37/xDY78a1vH49Q=; + b=gIxScFvlbW9Wuti5xZpTjEk8Q3kLDdTCGfqZ17qMGcgKVa70ElhlJLq0hex0jFChZF + oqyIN5Klmv8vvmt0Sg8rs2uMDcGZhQiWM8fdtynmC3UWfs9pAXnrt9pThsGsAlmHjnUV + wUcfEy4kSLpTx+76F5lk/LiQwDgGRIcsUqZbA89bK+POA9SG6QJxxh7SbLQZNWvcvRAm + TVpzfr2XRB8e8XKEOjzq+U0dKuCpLaSJjxy6Rd7OsrUHVmxKp7HXKt7ff1+w8MwU7I1j + GfSv9cdv5iRz1c+EWExRkhiRONks6hX7dHAel2ym1w+1JqwS0VBpBl2YrLwF3pT03lH2 + CB9Q== +Received: by 10.194.82.2 with SMTP id e2mr2122285wjy.39.1355250402531; + Tue, 11 Dec 2012 10:26:42 -0800 (PST) +Received: from localhost (cpc14-sgyl28-2-0-cust332.18-2.cable.virginmedia.com. + [82.39.85.77]) + by mx.google.com with ESMTPS id bd7sm17437419wib.8.2012.12.11.10.26.39 + (version=TLSv1/SSLv3 cipher=OTHER); + Tue, 11 Dec 2012 10:26:40 -0800 (PST) +Content-Type: multipart/signed; protocol="application/pgp-signature"; + micalg="pgp-sha1"; boundary="===============7418330742985369319==" +MIME-Version: 1.0 +Content-Disposition: inline +To: "Jason A. Donenfeld" , +From: Patrick Totzke +In-Reply-To: + +References: + +Message-ID: <20121211182638.27237.98903@brick.lan> +User-Agent: alot/0.3.3+ +Subject: Re: notmuch python bindings corrupt db index (was: gmail importer + script) +Date: Tue, 11 Dec 2012 18:26:38 +0000 +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 11 Dec 2012 18:26:47 -0000 + +--===============7418330742985369319== +Content-Type: text/plain; charset="utf-8" +MIME-Version: 1.0 +Content-Transfer-Encoding: quoted-printable + +Quoting Jason A. Donenfeld (2012-12-11 13:47:32) +> On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke +> wrote: +> > I ended up killing the process after i saw that ETA was >3h. +> = + +> Gmail's bottleneck. Very unfortunate. Leave it going over night. + +OK, i assumed i'd not have to download all messages because i already have = +them +on disk -- synced via offlineimap. but then i remembered that OI stores uses +its own naming scheme for the files in a maildir. so thats probably why you= +r script +did not realize theyre already there. (maybe it should? it could look up th= +e MIDs from notmuch) + +> > This broke my index! +> > +> > afterwards, a `notmuch new` printed +> > +> > A Xapian exception occurred creating a directory: Expected block +> > 485 to be level 1, not 0. +> > +> > I wasnt able to properly read the index anymore and had to +> > restore an old index dump. not cool! +> = + +> = + +> Can you provide any more information about this? The script properly +> uses atomic transactions. Any corruption here is the result of a +> notmuch library bug, which should be investigated. + +i am kind of reluctant to reproduce this again (on my work-desktop). +All i did was start your script, writing to the index in another terminal i= +n between (using alot) +possibly my cronjob fired a "notmuch new; afew.." in between also. +when I killed your script the index war broken. + +Can i turn on some global logging for notmuch to debug this? +Maybe libnotmuch had trouble with so many db-writes at a time? + +/p + + + +--===============7418330742985369319== +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Content-Description: signature +Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii" + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.11 (GNU/Linux) + +iEYEABECAAYFAlDHet4ACgkQlDQDZ9fWxap4BgCdFxBrgs+jIadzi/xvsAVB9mnY +s2UAoJZXSab5ftLEa1+d7ORHFiHsJy62 +=WT0i +-----END PGP SIGNATURE----- + +--===============7418330742985369319==--