Date: Sat, 15 Dec 2012 01:18:06 -0500
From: Austin Clements <amdragon@MIT.EDU>
To: "Jason A. Donenfeld" <Jason@zx2c4.com>
Subject: Re: notmuch python bindings corrupt db index (was: gmail importer
	script)
Message-ID: <20121215061806.GF6187@mit.edu>
References:
 <CAHmME9phAD0JKHDf+1KuCPyo4WXpDW5NESAE_iT3Ruv3bEVCuw@mail.gmail.com>
	<20121211182638.27237.98903@brick.lan>
	<CAHmME9qafP5He4hYPEAyrtS4hq3CRNKBQfvxvP=r4L6uDeqt1w@mail.gmail.com>
	<20121212204922.GB6187@mit.edu>
	<CAHmME9q-NbhRC=Be20aW5Vz+swou419zJxZD4O-3PSTJr+K=iQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To:
 <CAHmME9q-NbhRC=Be20aW5Vz+swou419zJxZD4O-3PSTJr+K=iQ@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: notmuch@notmuchmail.org
Precedence: list

Quoth Jason A. Donenfeld on Dec 13 at  3:32 pm:
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements <amdragon@mit.edu> wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.
> 
> 
> Austin-- think you could do a quick review of the script to double
> check and confirm I'm not doing anything nefarious?
> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

In theory the only way you could cause corruption besides tickling a
bug would be to access the same database object concurrently from
different threads (since it's not thread-safe), but you don't appear
to be doing that.

I did spot something that could corrupt delivered email, though.  The
way you deliver to the Maildir is resilient to process termination,
but not to system failures such as power outages.  In particular, you
need to at least os.fsync before the os.link.  I'd recommend looking
at Python's mailbox module, which has a robust Maildir delivery
implementation (though it appears it doesn't let you control the file
name, so you probably can't use it directly).