From 781ace6ac0edbdb3766d8440b11aaf6dad01d37f Mon Sep 17 00:00:00 2001 From: Austin Clements Date: Thu, 19 Apr 2012 13:54:50 +2000 Subject: [PATCH] Re: [RFC] Split notmuch_database_close into two functions --- d9/948cc5f07025bbe61945b0778dfbeb1d018ada | 152 ++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 d9/948cc5f07025bbe61945b0778dfbeb1d018ada diff --git a/d9/948cc5f07025bbe61945b0778dfbeb1d018ada b/d9/948cc5f07025bbe61945b0778dfbeb1d018ada new file mode 100644 index 000000000..55387a6d8 --- /dev/null +++ b/d9/948cc5f07025bbe61945b0778dfbeb1d018ada @@ -0,0 +1,152 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 1B611431FB6 + for ; Wed, 18 Apr 2012 10:54:56 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id zq9YFreSjHto for ; + Wed, 18 Apr 2012 10:54:55 -0700 (PDT) +Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU + [18.9.25.15]) + by olra.theworths.org (Postfix) with ESMTP id 4F664431FAE + for ; Wed, 18 Apr 2012 10:54:55 -0700 (PDT) +X-AuditID: 1209190f-b7f8a6d000000914-f3-4f8effeda258 +Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) + by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP + id 4D.F3.02324.DEFFE8F4; Wed, 18 Apr 2012 13:54:54 -0400 (EDT) +Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) + by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id q3IHsqrG012012; + Wed, 18 Apr 2012 13:54:53 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q3IHsojh017006 + (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); + Wed, 18 Apr 2012 13:54:51 -0400 (EDT) +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77) + (envelope-from ) + id 1SKZ5e-0001EK-Df; Wed, 18 Apr 2012 13:54:50 -0400 +Date: Wed, 18 Apr 2012 13:54:50 -0400 +From: Austin Clements +To: Mark Walters +Subject: Re: [RFC] Split notmuch_database_close into two functions +Message-ID: <20120418175450.GR13549@mit.edu> +References: + <1332291311-28954-1-git-send-email-4winter@informatik.uni-hamburg.de> + <20120401032323.GH5949@mit.edu> + <20120412090533.2074.78211@thinkbox.jade-hamburg.de> + <20120412165744.GF13549@mit.edu> <87mx6a4uls.fsf@qmul.ac.uk> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <87mx6a4uls.fsf@qmul.ac.uk> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFuphleLIzCtJLcpLzFFi42IR4hTV1n33v8/f4PwNJovZrT+YLFbP5bG4 + fnMmswOzx85Zd9k9Jp4/zebxbNUt5gDmKC6blNSczLLUIn27BK6MOz/fshf8UKj43XKNrYFx + q1QXIyeHhICJxKR/3UwQtpjEhXvr2boYuTiEBPYxSsxavIkdwtnAKLH73W9GCOckk8TFtyvZ + QVqEBJYwSrxsjAGxWQRUJW7N2MAGYrMJaEhs27+cEcQWEdCRuH1oAVg9s4CfxL0rV1lAbGEB + J4k1FyYBxTk4eIFqWtemQ8x/zSjRfe0JWA2vgKDEyZkQNrOAlsSNfy+ZQOqZBaQllv/jAAlz + Aq06d3UiK4gtKqAiMeXkNrYJjEKzkHTPQtI9C6F7ASPzKkbZlNwq3dzEzJzi1GTd4uTEvLzU + Il0TvdzMEr3UlNJNjOBAl+TfwfjtoNIhRgEORiUe3s/X+vyFWBPLiitzDzFKcjApifJe/wsU + 4kvKT6nMSCzOiC8qzUktPsQowcGsJMK77gJQjjclsbIqtSgfJiXNwaIkzqum9c5PSCA9sSQ1 + OzW1ILUIJivDwaEkwVsOjGghwaLU9NSKtMycEoQ0EwcnyHAeoOEFIDW8xQWJucWZ6RD5U4yK + UuK8biAJAZBERmkeXC8sEb1iFAd6RZg3AaSKB5jE4LpfAQ1mAhqsKAE2uCQRISXVwMikMCci + pC9D9F/qedY7+/8K69868iqZ72he++sTpSvLPkWsFmbv1Ak3dbItCC3fmbDocLstxzL+BD+J + LrU5fHefs0+ZwvAuImOqBE/XvwmbOtd8WSGZeVF+QsqeOWuFr1fPfNC5uf3YnmnKvn0Zb8qL + rJPjdSXLKxLOstUIF+91uxPocj37rhJLcUaioRZzUXEiAAej/zcfAwAA +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Wed, 18 Apr 2012 17:54:56 -0000 + +Quoth Mark Walters on Apr 17 at 9:42 am: +> On Thu, 12 Apr 2012, Austin Clements wrote: +> > Quoth Justus Winter on Apr 12 at 11:05 am: +> >> Quoting Austin Clements (2012-04-01 05:23:23) +> >> >Quoth Justus Winter on Mar 21 at 1:55 am: +> >> >> I propose to split the function notmuch_database_close into +> >> >> notmuch_database_close and notmuch_database_destroy so that long +> >> >> running processes like alot can close the database while still using +> >> >> data obtained from queries to that database. +> >> > +> >> >Is this actually safe? My understanding of Xapian::Database::close is +> >> >that, once you've closed the database, basically anything can throw a +> >> >Xapian exception. A lot of data is retrieved lazily, both by notmuch +> >> >and by Xapian, so simply having, say, a notmuch_message_t object isn't +> >> >enough to guarantee that you'll be able to get data out of it after +> >> >closing the database. Hence, I don't see how this interface could be +> >> >used correctly. +> >> +> >> I do not know how, but both alot and afew (and occasionally the +> >> notmuch binary) are somehow safely using this interface on my box for +> >> the last three weeks. +> > +> > I see. TL;DR: This isn't safe, but that's okay if we document it. +> > +> > The bug report [0] you pointed to was quite informative. At its core, +> > this is really a memory management issue. To sum up for the record +> > (and to check my own thinking): It sounds like alot is careful not to +> > use any notmuch objects after closing the database. The problem is +> > that, currently, closing the database also talloc_free's it, which +> > recursively free's everything derived from it. Python later GCs the +> > wrapper objects, which *also* try to free their underlying objects, +> > resulting in a double free. +> > +> > Before the change to expose notmuch_database_close, the Python +> > bindings would only talloc_free from destructors. Furthermore, they +> > prevented the library from recursively freeing things at other times +> > by internally maintaining a reverse reference for every library talloc +> > reference (e.g., message is a sub-allocation of query, so the bindings +> > keep a reference from each message to its query to ensure the query +> > doesn't get freed). The ability to explicitly talloc_free the +> > database subverts this mechanism. +> > +> > +> > So, I've come around to thinking that splitting notmuch_database_close +> > and _destroy is okay. It certainly parallels the rest of the API +> > better. However, notmuch_database_close needs a big warning similar +> > to Xapian::Database::close's warning that retrieving information from +> > objects derived from this database may not work after calling close. +> > notmuch_database_close is really a specialty interface, and about the +> > only thing you can guarantee after closing the database is that you +> > can destroy other objects. This is also going to require a SONAME +> > major version bump, as mentioned by others. Which, to be fair, would +> > be a good opportunity to fix some other issues, too, like how +> > notmuch_database_open can't return errors and how +> > notmuch_database_get_directory is broken on read-only databases. The +> > actual bump should be done at release time, but maybe we should drop a +> > note somewhere (NEWS?) so we don't forget. +> +> Can I just check that there is no way to reopen the Xapian database +> readonly? (I may be using the wrong term: I mean is there a way of +> switching an open read-write database to read-only without losing the +> attached structures/messages/threads etc) If I understand it this would +> be sufficient as it would free the lock, but could be more generally +> useful for long lived notmuch processes. + +That would be handy and perfect for this situation, but no (I +double-checked with Olly on IRC, which you probably saw). We might be +able to lobby for this capability if it seems more generally useful. +On the other hand, I think it would probably mix poorly with Xapian's +optimistic snapshot isolation if we tried to use it for anything +non-trivial (combined with real snapshot isolation it would be +awesome). -- 2.26.2