From: Austin Clements Date: Thu, 12 Apr 2012 16:57:44 +0000 (+2000) Subject: Re: [RFC] Split notmuch_database_close into two functions X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=a7454273b5b0cf20621d2d5ae7b30c2aa019cffc;p=notmuch-archives.git Re: [RFC] Split notmuch_database_close into two functions --- diff --git a/f3/12cca46a53840b06487263a5d169cfaa3d6e41 b/f3/12cca46a53840b06487263a5d169cfaa3d6e41 new file mode 100644 index 000000000..1e94501f2 --- /dev/null +++ b/f3/12cca46a53840b06487263a5d169cfaa3d6e41 @@ -0,0 +1,136 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id B170A431FAF + for ; Thu, 12 Apr 2012 09:57:50 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id Ks05ixbL1F5s for ; + Thu, 12 Apr 2012 09:57:50 -0700 (PDT) +Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU + [18.9.25.15]) + by olra.theworths.org (Postfix) with ESMTP id E21AC431FAE + for ; Thu, 12 Apr 2012 09:57:49 -0700 (PDT) +X-AuditID: 1209190f-b7f8a6d000000914-bb-4f87098b16b6 +Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) + by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP + id 12.0B.02324.B89078F4; Thu, 12 Apr 2012 12:57:47 -0400 (EDT) +Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) + by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id q3CGvkWV021095; + Thu, 12 Apr 2012 12:57:47 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q3CGviOd024327 + (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); + Thu, 12 Apr 2012 12:57:45 -0400 (EDT) +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77) + (envelope-from ) + id 1SINL6-0004IU-2r; Thu, 12 Apr 2012 12:57:44 -0400 +Date: Thu, 12 Apr 2012 12:57:44 -0400 +From: Austin Clements +To: Justus Winter <4winter@informatik.uni-hamburg.de> +Subject: Re: [RFC] Split notmuch_database_close into two functions +Message-ID: <20120412165744.GF13549@mit.edu> +References: + <1332291311-28954-1-git-send-email-4winter@informatik.uni-hamburg.de> + <20120401032323.GH5949@mit.edu> + <20120412090533.2074.78211@thinkbox.jade-hamburg.de> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <20120412090533.2074.78211@thinkbox.jade-hamburg.de> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFmphleLIzCtJLcpLzFFi42IR4hRV1u3mbPc3uDFZ32J26w8mi+s3ZzI7 + MHlMPH+azePZqlvMAUxRXDYpqTmZZalF+nYJXBl/Ln1jLNgpUfH2u1cDY4NIFyMnh4SAicTc + tl5mCFtM4sK99WxdjFwcQgL7GCVu3XkI5WxglDhyfSY7hHOSSeLezqVMEM4SRonmnd3sIP0s + AqoSl59tYQWx2QQ0JLbtX84IYosImEpsePAArIZZQFri2+9mJhBbWMBJYs2FSWBxXgEdiWln + vkINXcUoMW3Tb6iEoMTJmU9YIJq1JG78ewlUxAE2aPk/DpAwp4CjxOOek2B7RQVUJKac3MY2 + gVFoFpLuWUi6ZyF0L2BkXsUom5JbpZubmJlTnJqsW5ycmJeXWqRropebWaKXmlK6iREc2JL8 + Oxi/HVQ6xCjAwajEw/viWZu/EGtiWXFl7iFGSQ4mJVHe1ezt/kJ8SfkplRmJxRnxRaU5qcWH + GCU4mJVEeP88BirnTUmsrEotyodJSXOwKInzqmm98xMSSE8sSc1OTS1ILYLJynBwKEnwigMj + WEiwKDU9tSItM6cEIc3EwQkynAdoOC9IDW9xQWJucWY6RP4Uo6KUOO9PDqCEAEgiozQPrheW + eF4xigO9IswrBNLOA0xacN2vgAYzAQ3mUwC5urgkESEl1cBo8yDVwjU/fb6HVONxhSCHW58N + P7uaFur1GKhI+3WGv9FRCl8WMX8D4w3eax1d2cuLtz2bphCQGNeY7nnHuDLD8ihT96+Ct73z + Krg9b/1iOMVhmiK2eOeZRz8LVi399sPnzaNtTZbHLV9+YJgc/y0nI54nTlBp5ZcDbh3njRiv + fI503pY8/6gSS3FGoqEWc1FxIgAexVgYFwMAAA== +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 12 Apr 2012 16:57:50 -0000 + +Quoth Justus Winter on Apr 12 at 11:05 am: +> Quoting Austin Clements (2012-04-01 05:23:23) +> >Quoth Justus Winter on Mar 21 at 1:55 am: +> >> I propose to split the function notmuch_database_close into +> >> notmuch_database_close and notmuch_database_destroy so that long +> >> running processes like alot can close the database while still using +> >> data obtained from queries to that database. +> > +> >Is this actually safe? My understanding of Xapian::Database::close is +> >that, once you've closed the database, basically anything can throw a +> >Xapian exception. A lot of data is retrieved lazily, both by notmuch +> >and by Xapian, so simply having, say, a notmuch_message_t object isn't +> >enough to guarantee that you'll be able to get data out of it after +> >closing the database. Hence, I don't see how this interface could be +> >used correctly. +> +> I do not know how, but both alot and afew (and occasionally the +> notmuch binary) are somehow safely using this interface on my box for +> the last three weeks. + +I see. TL;DR: This isn't safe, but that's okay if we document it. + +The bug report [0] you pointed to was quite informative. At its core, +this is really a memory management issue. To sum up for the record +(and to check my own thinking): It sounds like alot is careful not to +use any notmuch objects after closing the database. The problem is +that, currently, closing the database also talloc_free's it, which +recursively free's everything derived from it. Python later GCs the +wrapper objects, which *also* try to free their underlying objects, +resulting in a double free. + +Before the change to expose notmuch_database_close, the Python +bindings would only talloc_free from destructors. Furthermore, they +prevented the library from recursively freeing things at other times +by internally maintaining a reverse reference for every library talloc +reference (e.g., message is a sub-allocation of query, so the bindings +keep a reference from each message to its query to ensure the query +doesn't get freed). The ability to explicitly talloc_free the +database subverts this mechanism. + + +So, I've come around to thinking that splitting notmuch_database_close +and _destroy is okay. It certainly parallels the rest of the API +better. However, notmuch_database_close needs a big warning similar +to Xapian::Database::close's warning that retrieving information from +objects derived from this database may not work after calling close. +notmuch_database_close is really a specialty interface, and about the +only thing you can guarantee after closing the database is that you +can destroy other objects. This is also going to require a SONAME +major version bump, as mentioned by others. Which, to be fair, would +be a good opportunity to fix some other issues, too, like how +notmuch_database_open can't return errors and how +notmuch_database_get_directory is broken on read-only databases. The +actual bump should be done at release time, but maybe we should drop a +note somewhere (NEWS?) so we don't forget. + +[0] https://github.com/pazz/alot/issues/413