--- /dev/null
+Return-Path: <patricktotzke@googlemail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 440E4429E28\r
+ for <notmuch@notmuchmail.org>; Thu, 26 May 2011 14:47:49 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.799\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,\r
+ FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id w7DehrVHubty for <notmuch@notmuchmail.org>;\r
+ Thu, 26 May 2011 14:47:48 -0700 (PDT)\r
+Received: from mail-ww0-f41.google.com (mail-ww0-f41.google.com\r
+ [74.125.82.41]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client\r
+ certificate requested) by olra.theworths.org (Postfix) with ESMTPS id\r
+ E223E431FB6 for <notmuch@notmuchmail.org>; Thu, 26 May 2011 14:47:47 -0700\r
+ (PDT)\r
+Received: by wwi18 with SMTP id 18so4809100wwi.2\r
+ for <notmuch@notmuchmail.org>; Thu, 26 May 2011 14:47:46 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+ d=googlemail.com; s=gamma;\r
+ h=domainkey-signature:cc:subject:from:to:in-reply-to:references:date\r
+ :message-id:user-agent:content-transfer-encoding:mime-version\r
+ :content-type; bh=Pyr3gAlEQ8tLu+6WfCLXTf6XAsWSdocNr9/tbJlIo08=;\r
+ b=Zu05y+x6JoTFAn/bvQR2FI0jVGrv3LIqGBzX1IVX8j0jtAHGG20bMNKK5QJd4LfkH4\r
+ VkPzWGWnUVL1p6GkkI2S4ypzZyIOO4cI08q12/8a78HRqYKt+dfW+bhdnIqzDZROqf2u\r
+ WmhI6xr+CdFP/ZQ32awY6465+ZABtpAeg6VEc=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma;\r
+ h=cc:subject:from:to:in-reply-to:references:date:message-id\r
+ :user-agent:content-transfer-encoding:mime-version:content-type;\r
+ b=JSJ/ruVrztAqV4hyscY/nnwNqktD+dyvNh0pAyIDX6PYr63XE7d+gRLH3adoHNufT6\r
+ V7/xvHbuHjBal4qVqijLJGcaKj0xfA14qit4CtdRImn7JyZ0QH6Pgsq5CB17e7K9qsUz\r
+ hjClP9hMftyfVlAM7h2gMoqvEvUzIhRXSIwaU=\r
+Received: by 10.227.54.6 with SMTP id o6mr1330767wbg.61.1306446466515;\r
+ Thu, 26 May 2011 14:47:46 -0700 (PDT)\r
+Received: from localhost (cpc1-sgyl2-0-0-cust47.sgyl.cable.virginmedia.com\r
+ [80.192.18.48])\r
+ by mx.google.com with ESMTPS id fm14sm761526wbb.58.2011.05.26.14.47.43\r
+ (version=TLSv1/SSLv3 cipher=OTHER);\r
+ Thu, 26 May 2011 14:47:45 -0700 (PDT)\r
+Subject: Re: one-time-iterators\r
+From: Patrick Totzke <patricktotzke@googlemail.com>\r
+To: Austin Clements <amdragon@mit.edu>\r
+In-reply-to: <BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>\r
+References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>\r
+ <BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>\r
+Date: Thu, 26 May 2011 22:47:42 +0100\r
+Message-Id: <1306446359-sup-9475@brick>\r
+User-Agent: Sup/git\r
+Content-Transfer-Encoding: 8bit\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; boundary="=-1306446462-339054-32122-5398-1-=";\r
+ protocol="application/pgp-signature"\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 26 May 2011 21:47:49 -0000\r
+\r
+\r
+--=-1306446462-339054-32122-5398-1-=\r
+Content-Type: text/plain; charset=UTF-8\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+hehe, did it again (dropping the list from cc). I need to stop\r
+using sup :P thanks Austin.\r
+\r
+Excerpts from Carl Worth's message of Thu May 26 18:20:21 +0100 2011:\r
+> On Thu, 26 May 2011 09:31:19 +0100, Patrick Totzke <patricktotzke@googl=\r
+email.com> wrote:\r
+> > Wow. This reads really complicated. All I want to say is:\r
+> > if I change tags in my search-results view, I get Xapian errors :)\r
+> =\r
+\r
+> Yes, that's frustrating. I wish that we had a more reliable interface a=\r
+t\r
+> the notmuch library level. But I'm not entirely sure what would be the\r
+> best way to do this.\r
+Actually, I expected something like this. For this reason each sup instan=\r
+ce =\r
+\r
+locks its index.\r
+At the moment I'm going for custom wrapper classes around notmuch.Thread\r
+and notmuch.Messages that cache the result of the calls relevant for me.\r
+But the real issue seems to be the iterator:\r
+It takes an awful lot of time just to copy the thread ids of all threads =\r
+from =\r
+\r
+large a query result.\r
+\r
+I tried the following in ipython:\r
+ q=3DDatabase().create_query('*')\r
+ time tids =3D [t.get_thread_id() for t in q.search_threads()]\r
+\r
+which results in\r
+CPU times: user 7.64 s, sys: 2.06 s, total: 9.70 s\r
+Wall time: 9.84 s\r
+\r
+It would really help if the Query object could return an iterator\r
+of thread-ids that makes this copying unnecessary. Is it possible to\r
+implement this? Or would this require the same amount of copying to happe=\r
+n\r
+at a lower level?\r
+I have not looked into the code for the bindings or the C code so far,\r
+but I guess the Query.search_threads() translates to some =\r
+\r
+"SELECT id,morestuff from threads"\r
+where for me a "SELECT is from threads" would totally suffice. Copying =\r
+\r
+(in the C code) only the ids so some list that yields an iterator should =\r
+be faster.\r
+\r
+> > The question: How do you solve this in the emacs code?\r
+> > do you store all tids of a query? =\r
+\r
+> =\r
+\r
+> The emacs code does not use the notmuch library interface like your\r
+> python bindings do. Instead, it uses the notmuch command-line tool, (an=\r
+d\r
+> buffers up the text output by it). =\r
+\r
+Ahh ok. Thanks for the explanation.\r
+\r
+Excerpts from Austin Clements's message of Thu May 26 21:18:53 +0100 2011=\r
+:\r
+> I proposed a solution to this problem a while ago\r
+> (id:"AANLkTi=3DKOx8aTJipkiArFVjEHE6zt_JypoASMiiAWBZ6@mail.gmail.com"),\r
+> though I haven't tried implementing it yet.\r
+Sorry, I wasn't on the list back then.\r
+\r
+> Though, Patrick, that solution doesn't address your problem.=C2=A0 On t=\r
+he\r
+> other hand, it's not clear to me what concurrent access semantics\r
+> you're actually expecting.=C2=A0 I suspect you don't want the remaining=\r
+\r
+> iteration to reflect the changes, since your changes could equally\r
+> well have affected earlier iteration results.=C2=A0\r
+That's right. =\r
+\r
+> But if you want a\r
+> consistent view of your query results, something's going to have to\r
+> materialize that iterator, and it might as well be you (or Xapian\r
+> would need more sophisticated concurrency control than it has).=C2=A0 B=\r
+ut\r
+> this shouldn't be expensive because all you need to materialize are\r
+> the document ids; you shouldn't need to eagerly fetch the per-thread\r
+> information. =\r
+\r
+I thought so, but it seems that Query.search_threads() already\r
+caches more than the id of each item. Which is as expected\r
+because it is designed to return thread objects, not their ids.\r
+As you can see above, this _is_ too expensive for me.\r
+\r
+> Have you tried simply calling list() on your thread\r
+> iterator to see how expensive it is? My bet is that it's quite cheap,\r
+> both memory-wise and CPU-wise.\r
+Funny thing:\r
+ q=3DDatabase().create_query('*')\r
+ time tlist =3D list(q.search_threads())\r
+raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some reason\r
+the list constructor must read mere than once from the iterator.\r
+So this is not an option, but even if it worked, it would show\r
+the same behaviour as my above test..\r
+\r
+would it be very hard to implement a Query.search_thread_ids() ?\r
+This name is a bit off because it had to be done on a lower level.\r
+Cheers,\r
+/p\r
+\r
+--=-1306446462-339054-32122-5398-1-=\r
+Content-Disposition: attachment; filename="signature.asc"\r
+Content-Type: application/pgp-signature; name="signature.asc"\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.11 (GNU/Linux)\r
+\r
+iEYEARECAAYFAk3eyn4ACgkQlDQDZ9fWxaqVKACeJYzPrEY/E60dnP0b9hBGJhvo\r
++VMAoK7XflpIiPibDDHdnRySul/LuQfP\r
+=Ez9a\r
+-----END PGP SIGNATURE-----\r
+\r
+--=-1306446462-339054-32122-5398-1-=--\r
projects / notmuch-archives.git / commitdiff