Re: [RFC] Split notmuch_database_close into two functions

diff --git a/d9/948cc5f07025bbe61945b0778dfbeb1d018ada b/d9/948cc5f07025bbe61945b0778dfbeb1d018ada
new file mode 100644 (file)
index 0000000..55387a6
--- /dev/null
+++ b/d9/948cc5f07025bbe61945b0778dfbeb1d018ada
@@ -0,0 +1,152 @@
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 1B611431FB6\r
+       for <notmuch@notmuchmail.org>; Wed, 18 Apr 2012 10:54:56 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id zq9YFreSjHto for <notmuch@notmuchmail.org>;\r
+       Wed, 18 Apr 2012 10:54:55 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU\r
+       [18.9.25.15])\r
+       by olra.theworths.org (Postfix) with ESMTP id 4F664431FAE\r
+       for <notmuch@notmuchmail.org>; Wed, 18 Apr 2012 10:54:55 -0700 (PDT)\r
+X-AuditID: 1209190f-b7f8a6d000000914-f3-4f8effeda258\r
+Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
+       by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP\r
+       id 4D.F3.02324.DEFFE8F4; Wed, 18 Apr 2012 13:54:54 -0400 (EDT)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+       by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id q3IHsqrG012012; \r
+       Wed, 18 Apr 2012 13:54:53 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+       (authenticated bits=0)\r
+       (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+       by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q3IHsojh017006\r
+       (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+       Wed, 18 Apr 2012 13:54:51 -0400 (EDT)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
+       (envelope-from <amdragon@mit.edu>)\r
+       id 1SKZ5e-0001EK-Df; Wed, 18 Apr 2012 13:54:50 -0400\r
+Date: Wed, 18 Apr 2012 13:54:50 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Mark Walters <markwalters1009@gmail.com>\r
+Subject: Re: [RFC] Split notmuch_database_close into two functions\r
+Message-ID: <20120418175450.GR13549@mit.edu>\r
+References:\r
+ <1332291311-28954-1-git-send-email-4winter@informatik.uni-hamburg.de>\r
+       <20120401032323.GH5949@mit.edu>\r
+       <20120412090533.2074.78211@thinkbox.jade-hamburg.de>\r
+       <20120412165744.GF13549@mit.edu> <87mx6a4uls.fsf@qmul.ac.uk>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <87mx6a4uls.fsf@qmul.ac.uk>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFuphleLIzCtJLcpLzFFi42IR4hTV1n33v8/f4PwNJovZrT+YLFbP5bG4\r
+       fnMmswOzx85Zd9k9Jp4/zebxbNUt5gDmKC6blNSczLLUIn27BK6MOz/fshf8UKj43XKNrYFx\r
+       q1QXIyeHhICJxKR/3UwQtpjEhXvr2boYuTiEBPYxSsxavIkdwtnAKLH73W9GCOckk8TFtyvZ\r
+       QVqEBJYwSrxsjAGxWQRUJW7N2MAGYrMJaEhs27+cEcQWEdCRuH1oAVg9s4CfxL0rV1lAbGEB\r
+       J4k1FyYBxTk4eIFqWtemQ8x/zSjRfe0JWA2vgKDEyZkQNrOAlsSNfy+ZQOqZBaQllv/jAAlz\r
+       Aq06d3UiK4gtKqAiMeXkNrYJjEKzkHTPQtI9C6F7ASPzKkbZlNwq3dzEzJzi1GTd4uTEvLzU\r
+       Il0TvdzMEr3UlNJNjOBAl+TfwfjtoNIhRgEORiUe3s/X+vyFWBPLiitzDzFKcjApifJe/wsU\r
+       4kvKT6nMSCzOiC8qzUktPsQowcGsJMK77gJQjjclsbIqtSgfJiXNwaIkzqum9c5PSCA9sSQ1\r
+       OzW1ILUIJivDwaEkwVsOjGghwaLU9NSKtMycEoQ0EwcnyHAeoOEFIDW8xQWJucWZ6RD5U4yK\r
+       UuK8biAJAZBERmkeXC8sEb1iFAd6RZg3AaSKB5jE4LpfAQ1mAhqsKAE2uCQRISXVwMikMCci\r
+       pC9D9F/qedY7+/8K69868iqZ72he++sTpSvLPkWsFmbv1Ak3dbItCC3fmbDocLstxzL+BD+J\r
+       LrU5fHefs0+ZwvAuImOqBE/XvwmbOtd8WSGZeVF+QsqeOWuFr1fPfNC5uf3YnmnKvn0Zb8qL\r
+       rJPjdSXLKxLOstUIF+91uxPocj37rhJLcUaioRZzUXEiAAej/zcfAwAA\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 18 Apr 2012 17:54:56 -0000\r
+\r
+Quoth Mark Walters on Apr 17 at  9:42 am:\r
+> On Thu, 12 Apr 2012, Austin Clements <amdragon@MIT.EDU> wrote:\r
+> > Quoth Justus Winter on Apr 12 at 11:05 am:\r
+> >> Quoting Austin Clements (2012-04-01 05:23:23)\r
+> >> >Quoth Justus Winter on Mar 21 at  1:55 am:\r
+> >> >> I propose to split the function notmuch_database_close into\r
+> >> >> notmuch_database_close and notmuch_database_destroy so that long\r
+> >> >> running processes like alot can close the database while still using\r
+> >> >> data obtained from queries to that database.\r
+> >> >\r
+> >> >Is this actually safe?  My understanding of Xapian::Database::close is\r
+> >> >that, once you've closed the database, basically anything can throw a\r
+> >> >Xapian exception.  A lot of data is retrieved lazily, both by notmuch\r
+> >> >and by Xapian, so simply having, say, a notmuch_message_t object isn't\r
+> >> >enough to guarantee that you'll be able to get data out of it after\r
+> >> >closing the database.  Hence, I don't see how this interface could be\r
+> >> >used correctly.\r
+> >> \r
+> >> I do not know how, but both alot and afew (and occasionally the\r
+> >> notmuch binary) are somehow safely using this interface on my box for\r
+> >> the last three weeks.\r
+> >\r
+> > I see.  TL;DR: This isn't safe, but that's okay if we document it.\r
+> >\r
+> > The bug report [0] you pointed to was quite informative.  At its core,\r
+> > this is really a memory management issue.  To sum up for the record\r
+> > (and to check my own thinking): It sounds like alot is careful not to\r
+> > use any notmuch objects after closing the database.  The problem is\r
+> > that, currently, closing the database also talloc_free's it, which\r
+> > recursively free's everything derived from it.  Python later GCs the\r
+> > wrapper objects, which *also* try to free their underlying objects,\r
+> > resulting in a double free.\r
+> >\r
+> > Before the change to expose notmuch_database_close, the Python\r
+> > bindings would only talloc_free from destructors.  Furthermore, they\r
+> > prevented the library from recursively freeing things at other times\r
+> > by internally maintaining a reverse reference for every library talloc\r
+> > reference (e.g., message is a sub-allocation of query, so the bindings\r
+> > keep a reference from each message to its query to ensure the query\r
+> > doesn't get freed).  The ability to explicitly talloc_free the\r
+> > database subverts this mechanism.\r
+> >\r
+> >\r
+> > So, I've come around to thinking that splitting notmuch_database_close\r
+> > and _destroy is okay.  It certainly parallels the rest of the API\r
+> > better.  However, notmuch_database_close needs a big warning similar\r
+> > to Xapian::Database::close's warning that retrieving information from\r
+> > objects derived from this database may not work after calling close.\r
+> > notmuch_database_close is really a specialty interface, and about the\r
+> > only thing you can guarantee after closing the database is that you\r
+> > can destroy other objects.  This is also going to require a SONAME\r
+> > major version bump, as mentioned by others.  Which, to be fair, would\r
+> > be a good opportunity to fix some other issues, too, like how\r
+> > notmuch_database_open can't return errors and how\r
+> > notmuch_database_get_directory is broken on read-only databases.  The\r
+> > actual bump should be done at release time, but maybe we should drop a\r
+> > note somewhere (NEWS?) so we don't forget.\r
+> \r
+> Can I just check that there is no way to reopen the Xapian database\r
+> readonly? (I may be using the wrong term: I mean is there a way of\r
+> switching an open read-write database to read-only without losing the\r
+> attached structures/messages/threads etc) If I understand it this would\r
+> be sufficient as it would free the lock, but could be more generally\r
+> useful for long lived notmuch processes.\r
+\r
+That would be handy and perfect for this situation, but no (I\r
+double-checked with Olly on IRC, which you probably saw).  We might be\r
+able to lobby for this capability if it seems more generally useful.\r
+On the other hand, I think it would probably mix poorly with Xapian's\r
+optimistic snapshot isolation if we tried to use it for anything\r
+non-trivial (combined with real snapshot isolation it would be\r
+awesome).\r