--- /dev/null
+Return-Path: <amdragon@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 86C2B431FB6\r
+ for <notmuch@notmuchmail.org>; Fri, 28 Jan 2011 08:50:10 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.698\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.698 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001,\r
+ HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=unavailable\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id wlToYhUUJdzS for <notmuch@notmuchmail.org>;\r
+ Fri, 28 Jan 2011 08:50:10 -0800 (PST)\r
+Received: from mail-qy0-f181.google.com (mail-qy0-f181.google.com\r
+ [209.85.216.181]) (using TLSv1 with cipher RC4-MD5 (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 3D579431FB5\r
+ for <notmuch@notmuchmail.org>; Fri, 28 Jan 2011 08:50:10 -0800 (PST)\r
+Received: by qyk12 with SMTP id 12so3847842qyk.5\r
+ for <notmuch@notmuchmail.org>; Fri, 28 Jan 2011 08:50:07 -0800 (PST)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;\r
+ h=domainkey-signature:mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type;\r
+ bh=LWh/YPZ3sZgb09PbtoIAIx20cRDOtthuwZx6kV5uadE=;\r
+ b=O14s8K51mdGbAh4cL+dDNyRol7NC5pQtTQ/gzw0RsKyjosspm2zDqqRKLTp7wWrm+9\r
+ yMb1+z/0PVR1VMAf8kXVXXykZW7sk5kr9ttCp9tpS5asrQcEe3pAf7dGJs/NB6TKAULx\r
+ 77Uf0foxxV1NYlCLWPEP8b6aIQ12/ofbWgx9E=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;\r
+ h=mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type;\r
+ b=GooC66BodM9j93SQvezKhlWQLUTsn/gQAk+tOd3QHAET10q72D+LMQ7+tXHFgRq2gl\r
+ grDzJxiM7YoJdIeSCiGLF1V05Mj4PC5lBEBjenNPrVsImJEhPiNSm4eJrtmZJ3ZwXsG7\r
+ PEHwCl8l7bq5tT344PPiyf3HOUPz8y/eGGPrs=\r
+MIME-Version: 1.0\r
+Received: by 10.229.220.81 with SMTP id hx17mr1097268qcb.116.1296233407012;\r
+ Fri, 28 Jan 2011 08:50:07 -0800 (PST)\r
+Sender: amdragon@gmail.com\r
+Received: by 10.229.97.143 with HTTP; Fri, 28 Jan 2011 08:50:06 -0800 (PST)\r
+In-Reply-To: <877hdps6og.fsf@kepler.schwinge.homeip.net>\r
+References: <87fwsetdin.fsf@kepler.schwinge.homeip.net>\r
+ <8762taxk9y.fsf@algae.riseup.net>\r
+ <87vd1a84qj.fsf@servo.finestructure.net>\r
+ <AANLkTi=pPgcPXUN_XTXa0gAkbTR_OF4XxCP-d4-Be2pt@mail.gmail.com>\r
+ <87fwsdobpy.fsf@yoom.home.cworth.org>\r
+ <877hdps6og.fsf@kepler.schwinge.homeip.net>\r
+Date: Fri, 28 Jan 2011 11:50:06 -0500\r
+X-Google-Sender-Auth: 78sJDERwTTQySqnTPb7GeuBDKhg\r
+Message-ID: <AANLkTinUhmcNKH6eFfcfygrc3XcVZoeDzVMFBSUi4LRQ@mail.gmail.com>\r
+Subject: Re: notmuch's idea of concurrency / failing an invocation\r
+From: Austin Clements <amdragon@mit.edu>\r
+To: Thomas Schwinge <thomas@schwinge.name>\r
+Content-Type: multipart/alternative; boundary=00163630eb23c56d45049aeadbb9\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 28 Jan 2011 16:50:10 -0000\r
+\r
+--00163630eb23c56d45049aeadbb9\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+\r
+On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge <thomas@schwinge.name>wrote:\r
+\r
+> On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth <cworth@cworth.org> wrote:\r
+> > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <amdragon@mit.edu>\r
+> wrote:\r
+> > > I'm looking into breaking notmuch new up into small transactions. It\r
+> > > wouldn't be much a leap from there to simply close and reopen the\r
+> database\r
+> > > between transactions if another task wants to use it, which would\r
+> release\r
+> > > the lock and let the queued notmuch task have the database for a bit.\r
+> >\r
+> > That sounds like something very useful to pursue. Please continue!\r
+>\r
+> Ack! And actually -- I just wondered about that: what happens if\r
+> ``notmuch new'' has executed notmuch_database_add_message for a new\r
+> message M, but then is killed before adding any tags and finishing up\r
+> (and supposing that the DB isn't in an invalid state now). This process\r
+> of adding M to the DB and applying any tags isn't one single transaction\r
+> currently, right? (And this is exactly what you're working on\r
+> chainging?) Am I right that what currently happens is that upon the next\r
+> ``notmuch new'' run, notmuch will not reconsider M (given that it already\r
+> is present in the DB), but continue with the next messages -- thus\r
+> leaving M without any tags? This isn't a very likely scenario, but still\r
+> a possible one.\r
+\r
+\r
+There are quite a few bugs like this. In fact, last night I added a test\r
+that interrupts notmuch new (for real, not SIGINT) after every database\r
+write, and on each interrupted database snapshot, re-runs notmuch new to\r
+completion, then checks that the database winds up in the correct state.\r
+ There are dozens of interruption points where it doesn't, many of which are\r
+permanent, even if you force notmuch new to rescan the maildir.\r
+\r
+> Another advantage that can happen with queueing (wherever it occurs) is\r
+> > to allow a client to be very responsive without waiting for an operation\r
+> > to complete. Though that can of course be band if the operation isn't\r
+> > reliably committed.\r
+>\r
+> (Obviously this can only work as long as we don't immediatelly need the\r
+> operation's result; think ``notmuch show''.)\r
+>\r
+> So, if the DB has the functionality to internally queue and immediatelly\r
+> acknowledge transactions, and only later (reliably) commit them, wouldn't\r
+> that be fine indeed? For example, ``notmuch tag'' then wouldn't have to\r
+> wait for the DB to be writable. (And note that I have no idea whether\r
+> Xapian supports such things.) But on the other hand we would like to\r
+> immediatelly display the requested change in the UI, right?\r
+>\r
+\r
+This would be fantastic, if the client could indicate the difference between\r
+a "pending" change and a "committed" change as you suggest below. I don't\r
+think having the database lie about its commit state is the right way to do\r
+this, though (nor should the client lie about this, thus the "pending"\r
+display). A better way would be for the client to update the display to\r
+"pending", start the notmuch operation asynchronously, have the notmuch\r
+operation block and queue up on the database lock, then have the client\r
+update the display to "committed" when the asynchronous operation returns.\r
+ No weird database operations or transactional semantics and the client side\r
+is fairly straightforward.\r
+\r
+--00163630eb23c56d45049aeadbb9\r
+Content-Type: text/html; charset=ISO-8859-1\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+<div class=3D"gmail_quote">On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge=\r
+ <span dir=3D"ltr"><<a href=3D"mailto:thomas@schwinge.name">thomas@schwi=\r
+nge.name</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=\r
+=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">\r
+<div class=3D"im">On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth <<a hre=\r
+f=3D"mailto:cworth@cworth.org">cworth@cworth.org</a>> wrote:<br>\r
+> On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <<a href=3D"mai=\r
+lto:amdragon@mit.edu">amdragon@mit.edu</a>> wrote:<br>\r
+> > I'm looking into breaking notmuch new up into small transacti=\r
+ons. =A0It<br>\r
+> > wouldn't be much a leap from there to simply close and reopen=\r
+ the database<br>\r
+> > between transactions if another task wants to use it, which would=\r
+ release<br>\r
+> > the lock and let the queued notmuch task have the database for a =\r
+bit.<br>\r
+><br>\r
+> That sounds like something very useful to pursue. Please continue!<br>\r
+<br>\r
+</div>Ack! =A0And actually -- I just wondered about that: what happens if<b=\r
+r>\r
+``notmuch new'' has executed notmuch_database_add_message for a new=\r
+<br>\r
+message M, but then is killed before adding any tags and finishing up<br>\r
+(and supposing that the DB isn't in an invalid state now). =A0This proc=\r
+ess<br>\r
+of adding M to the DB and applying any tags isn't one single transactio=\r
+n<br>\r
+currently, right? =A0(And this is exactly what you're working on<br>\r
+chainging?) =A0Am I right that what currently happens is that upon the next=\r
+<br>\r
+``notmuch new'' run, notmuch will not reconsider M (given that it a=\r
+lready<br>\r
+is present in the DB), but continue with the next messages -- thus<br>\r
+leaving M without any tags? =A0This isn't a very likely scenario, but s=\r
+till<br>\r
+a possible one.</blockquote><div>=A0</div><div>There are quite a few bugs l=\r
+ike this. =A0In fact, last night I added a test that interrupts notmuch new=\r
+=A0(for real, not SIGINT)=A0after every database write, and on each interru=\r
+pted database snapshot, re-runs notmuch new to completion, then checks that=\r
+ the database winds up in the correct state. =A0There are dozens of interru=\r
+ption points where it doesn't, many of which are permanent, even if you=\r
+ force notmuch new to rescan the maildir.</div>\r
+<meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><d=\r
+iv><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;b=\r
+order-left:1px #ccc solid;padding-left:1ex;"><div class=3D"im">> Another=\r
+ advantage that can happen with queueing (wherever it occurs) is</div>\r
+<div class=3D"im">\r
+> to allow a client to be very responsive without waiting for an operati=\r
+on<br>\r
+> to complete. Though that can of course be band if the operation isn=\r
+9;t<br>\r
+> reliably committed.<br>\r
+<br>\r
+</div>(Obviously this can only work as long as we don't immediatelly ne=\r
+ed the<br>\r
+operation's result; think ``notmuch show''.)<br>\r
+<br>\r
+So, if the DB has the functionality to internally queue and immediatelly<br=\r
+>\r
+acknowledge transactions, and only later (reliably) commit them, wouldn'=\r
+;t<br>\r
+that be fine indeed? =A0For example, ``notmuch tag'' then wouldn=\r
+9;t have to<br>\r
+wait for the DB to be writable. =A0(And note that I have no idea whether<br=\r
+>\r
+Xapian supports such things.) =A0But on the other hand we would like to<br>\r
+immediatelly display the requested change in the UI, right?<br></blockquote=\r
+><div><br></div><div>This would be fantastic, if the client could indicate =\r
+the difference between a "pending" change and a "committed&q=\r
+uot; change as you suggest below. =A0I don't think having the database =\r
+lie about its commit state is the right way to do this, though (nor should =\r
+the client lie about this, thus the "pending" display). =A0A bett=\r
+er way would be for the client to update the display to "pending"=\r
+, start the notmuch operation asynchronously, have the notmuch operation bl=\r
+ock and queue up on the database lock, then have the client update the disp=\r
+lay to "committed" when the asynchronous operation returns. =A0No=\r
+ weird database operations or transactional semantics and the client side i=\r
+s fairly straightforward.</div>\r
+<div><br></div></div>\r
+\r
+--00163630eb23c56d45049aeadbb9--\r
projects / notmuch-archives.git / commitdiff