Re: notmuch python bindings corrupt db index (was: gmail importer script)

diff --git a/aa/3fa50f7c429b51fa633377a6fd91e484736784 b/aa/3fa50f7c429b51fa633377a6fd91e484736784
new file mode 100644 (file)
index 0000000..2cdfcb9
--- /dev/null
+++ b/aa/3fa50f7c429b51fa633377a6fd91e484736784
@@ -0,0 +1,104 @@
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 3C627431FBC\r
+       for <notmuch@notmuchmail.org>; Wed, 12 Dec 2012 12:49:28 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id RFgI5+-oFAbI for <notmuch@notmuchmail.org>;\r
+       Wed, 12 Dec 2012 12:49:27 -0800 (PST)\r
+Received: from dmz-mailsec-scanner-3.mit.edu (DMZ-MAILSEC-SCANNER-3.MIT.EDU\r
+       [18.9.25.14])\r
+       by olra.theworths.org (Postfix) with ESMTP id AD9C7431FB6\r
+       for <notmuch@notmuchmail.org>; Wed, 12 Dec 2012 12:49:27 -0800 (PST)\r
+X-AuditID: 1209190e-b7f516d0000008e4-4d-50c8edd70b77\r
+Received: from mailhub-auth-1.mit.edu ( [18.9.21.35])\r
+       by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP\r
+       id 75.B0.02276.7DDE8C05; Wed, 12 Dec 2012 15:49:27 -0500 (EST)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+       by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id qBCKnQwm023632; \r
+       Wed, 12 Dec 2012 15:49:26 -0500\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+       (authenticated bits=0)\r
+       (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+       by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id qBCKnNrk024766\r
+       (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+       Wed, 12 Dec 2012 15:49:25 -0500 (EST)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+       (envelope-from <amdragon@mit.edu>)\r
+       id 1TitF4-0007I4-SE; Wed, 12 Dec 2012 15:49:22 -0500\r
+Date: Wed, 12 Dec 2012 15:49:22 -0500\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Patrick Totzke <patricktotzke@gmail.com>\r
+Subject: Re: notmuch python bindings corrupt db index (was: gmail importer\r
+       script)\r
+Message-ID: <20121212204922.GB6187@mit.edu>\r
+References:\r
+ <CAHmME9phAD0JKHDf+1KuCPyo4WXpDW5NESAE_iT3Ruv3bEVCuw@mail.gmail.com>\r
+       <20121211182638.27237.98903@brick.lan>\r
+       <CAHmME9qafP5He4hYPEAyrtS4hq3CRNKBQfvxvP=r4L6uDeqt1w@mail.gmail.com>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To:\r
+ <CAHmME9qafP5He4hYPEAyrtS4hq3CRNKBQfvxvP=r4L6uDeqt1w@mail.gmail.com>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFuphleLIzCtJLcpLzFFi42IR4hRV1r3+9kSAwfkV/BYPrkpZXL85k9ni\r
+       4hdZB2aPnbPusns8W3WL2WPy36fMAcxRXDYpqTmZZalF+nYJXBkbm26zFMzirNh57BNjA+NC\r
+       9i5GDg4JAROJBZvVuxg5gUwxiQv31rN1MXJxCAnsY5S4f3clC4SzgVGi83oDI4RzkUni3/lG\r
+       dghnCaPErSeHmUH6WQRUJZ4/fskIYrMJaEhs278czBYR0JF4uGAnE4jNLOAsMWnPbRYQW1gg\r
+       ROJldwtYnFdAW+Jjy0GoDQcYJa5MfwKVEJQ4OfMJC0SzlsSNfy+ZQO5mFpCWWP6PAyTMKRAo\r
+       8XjaKbAbRAVUJKac3MY2gVFoFpLuWUi6ZyF0L2BkXsUom5JbpZubmJlTnJqsW5ycmJeXWqRr\r
+       rJebWaKXmlK6iREc6JJ8Oxi/HlQ6xCjAwajEw3vh5fEAIdbEsuLK3EOMkhxMSqK8K5+cCBDi\r
+       S8pPqcxILM6ILyrNSS0+xCjBwawkwut2ACjHm5JYWZValA+TkuZgURLnvZJy019IID2xJDU7\r
+       NbUgtQgmK8PBoSTBqw6MaCHBotT01Iq0zJwShDQTByfIcB6g4dwgNbzFBYm5xZnpEPlTjIpS\r
+       4rzn3wAlBEASGaV5cL2wRPSKURzoFWFeDpB2HmASg+t+BTSYCWhw3KXjIINLEhFSUg2Mwj42\r
+       uc8P+zw4/qxd57NIsGPXTNOtlYUN65Jzv6rX+7DGShXVsGzlnPHj3MnpOQ96hMzeXLDcl3Qq\r
+       MVzHhSF/ukToNQ024wNn93i7H+mz32l1IKu4WeLQs7fcfffskidmu9nNVpIXPVkmlsWpe+zb\r
+       rrJ76fsccgvuHXYtuTErtm7ZT0GWrYZKLMUZiYZazEXFiQA9XRU0HwMAAA==\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 12 Dec 2012 20:49:28 -0000\r
+\r
+Quoth Jason A. Donenfeld on Dec 12 at  7:35 pm:\r
+> On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke <patricktotzke@gmail.com> wrote:\r
+> > i am kind of reluctant to reproduce this again (on my work-desktop).\r
+> \r
+> It'd really be a huge help if you could backup and try...\r
+> \r
+> > All i did was start your script, writing to the index in another terminal in between (using alot)\r
+> > possibly my cronjob fired a "notmuch new; afew.." in between also.\r
+> > when I killed your script the index war broken.\r
+> >\r
+> > Can i turn on some global logging for notmuch to debug this?\r
+> > Maybe libnotmuch had trouble with so many db-writes at a time?\r
+> \r
+> That is very wrong and bizarre. I open the database in read/write\r
+> mode, which is supposed to lock other programs from accessing it in\r
+> this way... Very strange.\r
+\r
+There should be no way to corrupt the database at this level through\r
+the Xapian API, which means nothing libnotmuch can do (much less users\r
+of libnotmuch) should be able to corrupt the database.  If you can\r
+reproduce the problem, it's probably a serious bug in Xapian, but it\r
+could also have been a file system bug or even random file system\r
+corruption.\r