--- /dev/null
+Return-Path: <amdragon@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 142D2429E28\r
+ for <notmuch@notmuchmail.org>; Thu, 26 May 2011 19:41:47 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.699\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.699 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001,\r
+ RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id dxVihK4TAvu5 for <notmuch@notmuchmail.org>;\r
+ Thu, 26 May 2011 19:41:45 -0700 (PDT)\r
+Received: from mail-qw0-f53.google.com (mail-qw0-f53.google.com\r
+ [209.85.216.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 0B09B431FB6\r
+ for <notmuch@notmuchmail.org>; Thu, 26 May 2011 19:41:44 -0700 (PDT)\r
+Received: by qwb7 with SMTP id 7so872055qwb.26\r
+ for <notmuch@notmuchmail.org>; Thu, 26 May 2011 19:41:44 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;\r
+ h=domainkey-signature:mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+ :content-transfer-encoding;\r
+ bh=phq5UPcAoJLmMxW0VGIZtZFLGrie0BSNhwo4E6f7q+Q=;\r
+ b=SxZfUTmEs9wMz62AJsxwe6OCrvMYzDXGcf7rYhjcUvZyVDQlwr1aipim1xxk12Dyvb\r
+ bH93yQ3dmy2zIfv4K7VRgl9b2Xy4NmEfSAQeAewFoq/6+fdcxdtO+Iu/69G2WrISLzkj\r
+ EshCH6bV5Dwh1/Lhy05DTHmjtn0WscbB7eagg=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;\r
+ h=mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+ :content-transfer-encoding;\r
+ b=EY9uFMcgEoV8ot9h43HjByz1PyeTIr4aZBm9epwqV3m3/5XWSaKI2aktU/YH5B4qtI\r
+ 0fKjKp1nf1BKN1giUbUkczjSVVbyHCkQ0wSCiz0nZv9IVpLWtoDf5tf/ENTJbcLux/Do\r
+ adAFfnG8XEL8sdvuc1KJhK4kOahDID7artTIk=\r
+MIME-Version: 1.0\r
+Received: by 10.229.35.1 with SMTP id n1mr1216915qcd.84.1306464104246; Thu, 26\r
+ May 2011 19:41:44 -0700 (PDT)\r
+Sender: amdragon@gmail.com\r
+Received: by 10.229.188.68 with HTTP; Thu, 26 May 2011 19:41:44 -0700 (PDT)\r
+In-Reply-To: <1306446621-sup-3184@brick>\r
+References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>\r
+ <BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>\r
+ <1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu>\r
+ <1306446621-sup-3184@brick>\r
+Date: Thu, 26 May 2011 22:41:44 -0400\r
+X-Google-Sender-Auth: lK5b05ERd-W7pUMIBxQrGZ3ex5Q\r
+Message-ID: <BANLkTi=Uk+bNB8sCZLVb86q-Kjfx1udEZA@mail.gmail.com>\r
+Subject: Re: one-time-iterators\r
+From: Austin Clements <amdragon@mit.edu>\r
+To: Patrick Totzke <patricktotzke@googlemail.com>\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+Content-Transfer-Encoding: quoted-printable\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 27 May 2011 02:41:47 -0000\r
+\r
+On Thu, May 26, 2011 at 6:22 PM, Patrick Totzke\r
+<patricktotzke@googlemail.com> wrote:\r
+> Excerpts from Austin Clements's message of Thu May 26 22:43:02 +0100 2011=\r
+:\r
+>> > > Though, Patrick, that solution doesn't address your problem.=A0 On t=\r
+he\r
+>> > > other hand, it's not clear to me what concurrent access semantics\r
+>> > > you're actually expecting.=A0 I suspect you don't want the remaining\r
+>> > > iteration to reflect the changes, since your changes could equally\r
+>> > > well have affected earlier iteration results.\r
+>> > That's right.\r
+>> > > But if you want a\r
+>> > > consistent view of your query results, something's going to have to\r
+>> > > materialize that iterator, and it might as well be you (or Xapian\r
+>> > > would need more sophisticated concurrency control than it has).=A0 B=\r
+ut\r
+>> > > this shouldn't be expensive because all you need to materialize are\r
+>> > > the document ids; you shouldn't need to eagerly fetch the per-thread\r
+>> > > information.\r
+>> > I thought so, but it seems that Query.search_threads() already\r
+>> > caches more than the id of each item. Which is as expected\r
+>> > because it is designed to return thread objects, not their ids.\r
+>> > As you can see above, this _is_ too expensive for me.\r
+>>\r
+>> I'd forgotten that constructing threads on the C side was eager about\r
+>> the thread tags, author list and subject (which, without Istvan's\r
+>> proposed patch, even requires opening and parsing the message file).\r
+>> This is probably what's killing you.\r
+>>\r
+>> Out of curiosity, what is your situation that you won't wind up paying\r
+>> the cost of this iteration one way or the other and that the latency\r
+>> of doing these tag changes matters?\r
+>\r
+> I'm trying to implement a terminal interface for notmuch in python\r
+> that resembles sup.\r
+> For the search results view, i read an initial portion from a Threads ite=\r
+rator\r
+> to fill my teminal window with threadline-widgets. Obviously, for a\r
+> large number of results I don't want to go through all of them.\r
+> The problem arises if you toggle a tag on the selected threadline and aft=\r
+erwards\r
+> continue to scroll down.\r
+\r
+Ah, that makes sense.\r
+\r
+>> > > Have you tried simply calling list() on your thread\r
+>> > > iterator to see how expensive it is? =A0My bet is that it's quite ch=\r
+eap,\r
+>> > > both memory-wise and CPU-wise.\r
+>> > Funny thing:\r
+>> > =A0q=3DDatabase().create_query('*')\r
+>> > =A0time tlist =3D list(q.search_threads())\r
+>> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some reas=\r
+on\r
+>> > the list constructor must read mere than once from the iterator.\r
+>> > So this is not an option, but even if it worked, it would show\r
+>> > the same behaviour as my above test..\r
+>>\r
+>> Interesting. =A0Looks like the Threads class implements __len__ and that\r
+>> its implementation exhausts the iterator. =A0Which isn't a great idea in\r
+>> itself, but it turns out that Python's implementation of list() calls\r
+>> __len__ if it's available (presumably to pre-size the list) before\r
+>> iterating over the object, so it exhausts the iterator before even\r
+>> using it.\r
+>>\r
+>> That said, if list(q.search_threads()) did work, it wouldn't give you\r
+>> better performance than your experiment above.\r
+>>\r
+>> > would it be very hard to implement a Query.search_thread_ids() ?\r
+>> > This name is a bit off because it had to be done on a lower level.\r
+>>\r
+>> Lazily fetching the thread metadata on the C side would probably\r
+>> address your problem automatically. =A0But what are you doing that\r
+>> doesn't require any information about the threads you're manipulating?\r
+> Agreed. Unfortunately, there seems to be no way to get a list of thread\r
+> ids or a reliable iterator thereof by using the current python bindings.\r
+> It would be enough for me to have the ids because then I could\r
+> search for the few threads I actually need individually on demand.\r
+\r
+There's no way to do that from the C API either, so don't feel left\r
+out. ]:--8) It seems to me that the right solution to your problem\r
+is to make thread information lazy (effectively, everything gathered\r
+in lib/thread.cc:_thread_add_message). Then you could probably\r
+materialize that iterator cheaply. In fact, it's probably worth\r
+trying a hack where you put dummy information in the thread object\r
+from _thread_add_message and see how long it takes just to walk the\r
+iterator (unfortunately I don't think profiling will help much here\r
+because much of your time is probably spent waiting for I/O).\r
+\r
+I don't think there would be any downside to doing this for eager\r
+consumers like the CLI.\r
projects / notmuch-archives.git / commitdiff