--- /dev/null
+Return-Path: <amdragon@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 5FE92431FD0\r
+ for <notmuch@notmuchmail.org>; Fri, 27 May 2011 12:29:26 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.699\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.699 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001,\r
+ RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id d7yxnsMy-thq for <notmuch@notmuchmail.org>;\r
+ Fri, 27 May 2011 12:29:25 -0700 (PDT)\r
+Received: from mail-qw0-f53.google.com (mail-qw0-f53.google.com\r
+ [209.85.216.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 3CBAD431FB6\r
+ for <notmuch@notmuchmail.org>; Fri, 27 May 2011 12:29:25 -0700 (PDT)\r
+Received: by qwb7 with SMTP id 7so1292064qwb.26\r
+ for <notmuch@notmuchmail.org>; Fri, 27 May 2011 12:29:24 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;\r
+ h=domainkey-signature:mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+ :content-transfer-encoding;\r
+ bh=kf3jqWmEfa0k0RzAXXx5a5GowCH/+zcPqUt7ehdLpNk=;\r
+ b=Wp3TBng+cHzs2GbMQynsAlu191+JBcBRAXuvtIYKhJrFES9vJhXuxUc7zXk7Ozr0Va\r
+ DzRO4gwXcwHPO6PDM2UjCtsd9qNCBty3MXXlHiGBZ1Wy8/SgMJMYv8+6DsHosjaBq8qS\r
+ 7WfnEQI32TvfrPJ9DJdHnJ6dvh1RY3CpC/gYw=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;\r
+ h=mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+ :content-transfer-encoding;\r
+ b=n/g+bv97+Xl/M31KEmPdBIsYyBj+TZx4SQkAd+YN3RFXUOZ+WHQ6zLJFg+24eWbWIP\r
+ vQgHfouEWmvCx7FwY3ug2qcAxrVWYdQT5T9G3vXZf7tbXzWVgil6BB4X/f/hvsXsl2ei\r
+ A1l+SUn9KsVSV8ZCm/bYzyOpB85euafJQqccM=\r
+MIME-Version: 1.0\r
+Received: by 10.229.43.99 with SMTP id v35mr1879992qce.8.1306524564220; Fri,\r
+ 27 May 2011 12:29:24 -0700 (PDT)\r
+Sender: amdragon@gmail.com\r
+Received: by 10.229.188.68 with HTTP; Fri, 27 May 2011 12:29:24 -0700 (PDT)\r
+In-Reply-To: <1306518628-sup-5396@brick>\r
+References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>\r
+ <BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>\r
+ <1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu>\r
+ <1306446621-sup-3184@brick>\r
+ <BANLkTi=Uk+bNB8sCZLVb86q-Kjfx1udEZA@mail.gmail.com>\r
+ <1306518628-sup-5396@brick>\r
+Date: Fri, 27 May 2011 15:29:24 -0400\r
+X-Google-Sender-Auth: UshyP3vImhrQ5skBKML1XvyreUY\r
+Message-ID: <BANLkTi=cZ50h50xf_OigTyjdfY_y34AX_g@mail.gmail.com>\r
+Subject: Re: one-time-iterators\r
+From: Austin Clements <amdragon@mit.edu>\r
+To: Patrick Totzke <patricktotzke@googlemail.com>,\r
+ Sebastian Spaeth <sebastian@sspaeth.de>\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+Content-Transfer-Encoding: quoted-printable\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 27 May 2011 19:29:26 -0000\r
+\r
+On Fri, May 27, 2011 at 2:04 PM, Patrick Totzke\r
+<patricktotzke@googlemail.com> wrote:\r
+> Excerpts from Austin Clements's message of Fri May 27 03:41:44 +0100 2011=\r
+:\r
+>> >> > > Have you tried simply calling list() on your thread\r
+>> >> > > iterator to see how expensive it is? =A0My bet is that it's quite=\r
+ cheap,\r
+>> >> > > both memory-wise and CPU-wise.\r
+>> >> > Funny thing:\r
+>> >> > =A0q=3DDatabase().create_query('*')\r
+>> >> > =A0time tlist =3D list(q.search_threads())\r
+>> >> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some r=\r
+eason\r
+>> >> > the list constructor must read mere than once from the iterator.\r
+>> >> > So this is not an option, but even if it worked, it would show\r
+>> >> > the same behaviour as my above test..\r
+>> >>\r
+>> >> Interesting. =A0Looks like the Threads class implements __len__ and t=\r
+hat\r
+>> >> its implementation exhausts the iterator. =A0Which isn't a great idea=\r
+ in\r
+>> >> itself, but it turns out that Python's implementation of list() calls\r
+>> >> __len__ if it's available (presumably to pre-size the list) before\r
+>> >> iterating over the object, so it exhausts the iterator before even\r
+>> >> using it.\r
+>> >>\r
+>> >> That said, if list(q.search_threads()) did work, it wouldn't give you\r
+>> >> better performance than your experiment above.\r
+> true. Nevertheless I think that list(q.search_threads())\r
+> should be equivalent to [t for t in q.search_threads()], which is\r
+> something to be fixed in the bindings. Should I file an issue somehow?\r
+> Or is enough to state this as a TODO here on the list?\r
+\r
+Yes, they should be equivalent.\r
+\r
+Sebastian was thinking about fixing the larger issue of generator\r
+exhaustion, which would address this, though the performance would\r
+depend on the cost of iterating twice. This is why generators\r
+shouldn't support __len__. Unfortunately, it's probably hard to get\r
+rid of at this point and I doubt there's a way to tell list() to\r
+overlook the presence of a __len__ method.\r
+\r
+>> >> > would it be very hard to implement a Query.search_thread_ids() ?\r
+>> >> > This name is a bit off because it had to be done on a lower level.\r
+>> >>\r
+>> >> Lazily fetching the thread metadata on the C side would probably\r
+>> >> address your problem automatically. =A0But what are you doing that\r
+>> >> doesn't require any information about the threads you're manipulating=\r
+?\r
+>> > Agreed. Unfortunately, there seems to be no way to get a list of threa=\r
+d\r
+>> > ids or a reliable iterator thereof by using the current python binding=\r
+s.\r
+>> > It would be enough for me to have the ids because then I could\r
+>> > search for the few threads I actually need individually on demand.\r
+>>\r
+>> There's no way to do that from the C API either, so don't feel left\r
+>> out. =A0]:--8) =A0It seems to me that the right solution to your problem\r
+>> is to make thread information lazy (effectively, everything gathered\r
+>> in lib/thread.cc:_thread_add_message). =A0Then you could probably\r
+>> materialize that iterator cheaply.\r
+> Alright. I'll put this on my mental notmuch wish list and\r
+> hope that someone will have addressed this before I run out of\r
+> ideas how to improve my UI and have time to look at this myself.\r
+> For now, I go with the [t.get_thread_id for t in q.search_threads()]\r
+> approach to cache the thread ids myself and live with the fact that\r
+> this takes time for large result sets.\r
+>\r
+>> In fact, it's probably worth\r
+>> trying a hack where you put dummy information in the thread object\r
+>> from _thread_add_message and see how long it takes just to walk the\r
+>> iterator (unfortunately I don't think profiling will help much here\r
+>> because much of your time is probably spent waiting for I/O).\r
+> I don't think I understand what you mean by dummy info in a thread\r
+> object.\r
+\r
+In _thread_add_message, rather than looking up the message's author,\r
+subject, etc, just hard-code some dummy values. Performance-wise,\r
+this would simulate making the thread metadata lookup lazy, so you\r
+could see if making this lazy would address your problem.\r
projects / notmuch-archives.git / commitdiff