From 31f946c165e8dfc7ca38944838783749ed92c118 Mon Sep 17 00:00:00 2001 From: Justus Winter <4winter@informatik.uni-hamburg.de> Date: Thu, 12 Apr 2012 19:19:39 +0200 Subject: [PATCH] Re: [RFC] Split notmuch_database_close into two functions --- 18/12405bbca1be073fc2767a8f83ef483ed0e4c1 | 128 ++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 18/12405bbca1be073fc2767a8f83ef483ed0e4c1 diff --git a/18/12405bbca1be073fc2767a8f83ef483ed0e4c1 b/18/12405bbca1be073fc2767a8f83ef483ed0e4c1 new file mode 100644 index 000000000..c2d0ba9d8 --- /dev/null +++ b/18/12405bbca1be073fc2767a8f83ef483ed0e4c1 @@ -0,0 +1,128 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 57402431FB6 + for ; Thu, 12 Apr 2012 10:20:49 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id DHZmjrBs27HI for ; + Thu, 12 Apr 2012 10:20:48 -0700 (PDT) +Received: from mail.cryptobitch.de (cryptobitch.de [88.198.7.68]) + (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 66748431FAF + for ; Thu, 12 Apr 2012 10:20:48 -0700 (PDT) +Received: from mail.jade-hamburg.de (unknown [85.183.11.228]) + (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by mail.cryptobitch.de (Postfix) with ESMTPSA id 9DCF356A61B + for ; Thu, 12 Apr 2012 19:20:46 +0200 (CEST) +Received: by mail.jade-hamburg.de (Postfix, from userid 401) + id 07F4FDF2A3; Thu, 12 Apr 2012 19:20:46 +0200 (CEST) +Received: from thinkbox.jade-hamburg.de (unknown [10.1.1.153]) + (using TLSv1 with cipher AES256-SHA (256/256 bits)) + (No client certificate requested) (Authenticated sender: teythoon) + by mail.jade-hamburg.de (Postfix) with ESMTPSA id 34EF2DF2A0; + Thu, 12 Apr 2012 19:20:43 +0200 (CEST) +Received: from teythoon by thinkbox.jade-hamburg.de with local (Exim 4.77) + (envelope-from ) + id 1SINgJ-0005ft-4i; Thu, 12 Apr 2012 19:19:39 +0200 +Content-Type: text/plain; charset="utf-8" +MIME-Version: 1.0 +Content-Transfer-Encoding: quoted-printable +To: Austin Clements , +From: Justus Winter <4winter@informatik.uni-hamburg.de> +In-Reply-To: <20120412165744.GF13549@mit.edu> +References: + <1332291311-28954-1-git-send-email-4winter@informatik.uni-hamburg.de> + <20120401032323.GH5949@mit.edu> + <20120412090533.2074.78211@thinkbox.jade-hamburg.de> + <20120412165744.GF13549@mit.edu> +Message-ID: <20120412171939.5852.22809@thinkbox.jade-hamburg.de> +User-Agent: alot/0.21+ +Subject: Re: [RFC] Split notmuch_database_close into two functions +Date: Thu, 12 Apr 2012 19:19:39 +0200 +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 12 Apr 2012 17:20:49 -0000 + +Quoting Austin Clements (2012-04-12 18:57:44) +>Quoth Justus Winter on Apr 12 at 11:05 am: +>> Quoting Austin Clements (2012-04-01 05:23:23) +>> >Quoth Justus Winter on Mar 21 at 1:55 am: +>> >> I propose to split the function notmuch_database_close into +>> >> notmuch_database_close and notmuch_database_destroy so that long +>> >> running processes like alot can close the database while still using +>> >> data obtained from queries to that database. +>> > +>> >Is this actually safe? My understanding of Xapian::Database::close is +>> >that, once you've closed the database, basically anything can throw a +>> >Xapian exception. A lot of data is retrieved lazily, both by notmuch +>> >and by Xapian, so simply having, say, a notmuch_message_t object isn't +>> >enough to guarantee that you'll be able to get data out of it after +>> >closing the database. Hence, I don't see how this interface could be +>> >used correctly. +>> = + +>> I do not know how, but both alot and afew (and occasionally the +>> notmuch binary) are somehow safely using this interface on my box for +>> the last three weeks. +> +>I see. TL;DR: This isn't safe, but that's okay if we document it. +> +>The bug report [0] you pointed to was quite informative. At its core, +>this is really a memory management issue. To sum up for the record +>(and to check my own thinking): It sounds like alot is careful not to +>use any notmuch objects after closing the database. The problem is +>that, currently, closing the database also talloc_free's it, which +>recursively free's everything derived from it. Python later GCs the +>wrapper objects, which *also* try to free their underlying objects, +>resulting in a double free. +> +>Before the change to expose notmuch_database_close, the Python +>bindings would only talloc_free from destructors. Furthermore, they +>prevented the library from recursively freeing things at other times +>by internally maintaining a reverse reference for every library talloc +>reference (e.g., message is a sub-allocation of query, so the bindings +>keep a reference from each message to its query to ensure the query +>doesn't get freed). The ability to explicitly talloc_free the +>database subverts this mechanism. + +Exactly. + +>So, I've come around to thinking that splitting notmuch_database_close +>and _destroy is okay. It certainly parallels the rest of the API +>better. However, notmuch_database_close needs a big warning similar +>to Xapian::Database::close's warning that retrieving information from +>objects derived from this database may not work after calling close. + +Yes, but then again one should always expect function calls to fail +and most APIs have mechanisms to communicate failures. + +OTOH this might be an indication that the notmuch API should be +redesigned. Both alot and afew have their own wrappers around the +notmuch API to work around some limitations (e.g. changes to messages +are enqueued and executed at some point, with some kind of mechanism +to cope with the notmuch database temporarily not being available, +message objects have to be re-fetched if they got outdated (IIRC, +whatever that means)). + +Justus -- 2.26.2