From: Gaute Hope Date: Thu, 21 Aug 2014 08:59:11 +0000 (+0200) Subject: Re: DatabaseModifiedErrors causing troubles X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=1cf689cf0a0f2371f64919decc7f0fcc47820aea;p=notmuch-archives.git Re: DatabaseModifiedErrors causing troubles --- diff --git a/eb/f8375f7ac6b88451429e2f7ef0d83d632e4296 b/eb/f8375f7ac6b88451429e2f7ef0d83d632e4296 new file mode 100644 index 000000000..4874c9af1 --- /dev/null +++ b/eb/f8375f7ac6b88451429e2f7ef0d83d632e4296 @@ -0,0 +1,164 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id D4EA4431FB6 + for ; Thu, 21 Aug 2014 01:58:59 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id u74j+oLJijAa for ; + Thu, 21 Aug 2014 01:58:50 -0700 (PDT) +Received: from mail-la0-f42.google.com (mail-la0-f42.google.com + [209.85.215.42]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 4CE73431FAF + for ; Thu, 21 Aug 2014 01:58:50 -0700 (PDT) +Received: by mail-la0-f42.google.com with SMTP id pv20so8377135lab.29 + for ; Thu, 21 Aug 2014 01:58:47 -0700 (PDT) +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=1e100.net; s=20130820; + h=x-gm-message-state:date:from:subject:to:references:in-reply-to + :user-agent:message-id:mime-version:content-type + :content-transfer-encoding; + bh=FxjRwAOO9beMJyXC9fBjjP4FY/NPHbGPl/peq+7npwE=; + b=S3X240f/Yq3GDLrLFQyT+RHQ4FJsImxGKnIFgio9v0Fd9AQOWeQiDIiJ5Kox5hRDj/ + tB8eprKSYm4Vh1SMr47e7VoKXTXrMqIp5UUDR+bkxUEGmzTKPkKrixjSM8jbIn0esB8u + HhJ9A2rNtIdPDZNixYpAGXOnzxI9C+thqUdyW71pBxDMrEmY6L6qXt+jDtfMS1l3Iyfw + DxFFybvl+dXEhhcXUan1JpmQrqUsyEkHYkHxIKVtaTMAKdZ45KfqZQF3uv0vpDH8dK9E + +9vbnvRSkPWWo8DqbHncei2Ntp5bgoaZZEerxX7sjwppFOwjGAmcNHHuQetqJKScjdf+ + GEvA== +X-Gm-Message-State: + ALoCoQn6Ew1NUatZ9EOFm1SXqYGsSs/5sutLGt/7T1qRGTTTLn/VuLhJ5FMsQWvs5+Me6xX8CpjE +X-Received: by 10.112.138.102 with SMTP id qp6mr44401999lbb.60.1408611527400; + Thu, 21 Aug 2014 01:58:47 -0700 (PDT) +Received: from localhost (cD572BF51.dhcp.as2116.net. [81.191.114.213]) + by mx.google.com with ESMTPSA id y8sm666676lad.30.2014.08.21.01.58.45 + for + (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); + Thu, 21 Aug 2014 01:58:46 -0700 (PDT) +Date: Thu, 21 Aug 2014 10:59:11 +0200 +From: Gaute Hope +Subject: Re: DatabaseModifiedErrors causing troubles +To: notmuch@notmuchmail.org +References: + +In-Reply-To: + +User-Agent: astroid/v3f318711 (https://github.com/gauteh/astroid) +Message-Id: <1408610770-astroid-0-4on2jlyj8n-28086@strange> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8; format=flowed +Content-Transfer-Encoding: quoted-printable +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 21 Aug 2014 08:59:00 -0000 + +Gaute Hope wrote on Mon, 11 Aug 2014 14:17:54 +0200: +> Hi, +> +> I've been working on an application that keeps a read-only handle on +> the notmuch database open for a long time. In some cases when a new +> message is added along with some renames of other messages using +> 'notmuch new' while the application is running I get an Xapian +> exception: DatabaseModifiedError: +> +> A Xapian exception occurred performing query: The revision being +> read has been discarded - you +> should call Xapian::Database::reopen() and retry the operation. +> +> Which seems to be printed from: notmuch_query_search_threads -> +> notmuch_query_search_messages:294. +> +> I have not been able to make a smaller test case at the moment (this +> happens with offlineimap updating an maildir and notmuch new run +> afterwards + some tagging). +> +> I can work around this by checking for a NULL pointer returned from +> notmuch_query_search_threads () and re-open the database +> (notmuch_database_close () -> notmuch_database_open ()). But I have no +> way of knowing programatically if this really is the error that has +> happened. There should be some way of propagating the error +> information or (even better for my case; for notmuch to reopen the +> database), one option is the Gmime way of passing an pointer to an +> error structure that is filled up by the notmuch interface function. + +Hi again, + +I have not yet been able to create a test for this with a fresh db (I +must be missing out on some of the parts triggering this). + +I need to keep a query open for a while (because running it takes a +while), and this keeps coming back and biting me. I currently do a test +every time I need to access the query further (load more threads..). The +test is as follows, which is the easiest and quickest test I've found to +work reliably: + + + /* tries to provoke an Xapian::DatabaseModifiedError + * + * returns true if db is invalid. queries will be invalid + * afterwards, but might still respond true to notmuch_valid + * calls. + * + */ + + time_t start =3D clock (); + bool invalid =3D false; + + /* testing */ + notmuch_query_t * q =3D notmuch_query_create (nm_db, "thread:fail"); + + /* this will produce an error, but no way to ensure failure */ + notmuch_query_count_threads (q); + + /* this returns NULL when error */ + notmuch_threads_t * threads =3D notmuch_query_search_threads (q); + + if (threads =3D=3D NULL) invalid =3D true; + + notmuch_query_destroy (q); + + float diff =3D (clock () - start) * 1000.0 / CLOCKS_PER_SEC; + cout << "db: test query time: " << diff << " ms." << endl; + + if (invalid) { + cout << "db: no longer valid, reopen required." << endl; + } else { + cout << "db: valid." << endl; + } + +As mentioned before, I am basically guessing that the reason I get +threads =3D=3D NULL is the DatabaseModifiedError. There is no way to know +for sure. + +For portability I would suggest starting to move towards the GError +scheme provided by glib (also used by gmime). This is a somewhat major +effort though since ensuring that error propagation is done right [0] is +somewhat tricky. It provides more or less the same functionality as +exceptions do in C++. + +There is also the problem of having to change the API - one way to avoid +that is to create wrappers that behave like the old API, but don't +handle errors that good. This requires the addition of addiontal _error +variations of the current set of functions. It will be a mess. + +Cheers, Gaute + +[0] https://developer.gnome.org/glib/stable/glib-Error-Reporting.html +=