DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:sender:in-reply-to:references:date
	:x-google-sender-auth:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	b=n/g+bv97+Xl/M31KEmPdBIsYyBj+TZx4SQkAd+YN3RFXUOZ+WHQ6zLJFg+24eWbWIP
	vQgHfouEWmvCx7FwY3ug2qcAxrVWYdQT5T9G3vXZf7tbXzWVgil6BB4X/f/hvsXsl2ei
	A1l+SUn9KsVSV8ZCm/bYzyOpB85euafJQqccM=
MIME-Version: 1.0
Sender: amdragon@gmail.com
In-Reply-To: <1306518628-sup-5396@brick>
References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>
	<BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>
	<1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu>
	<1306446621-sup-3184@brick>
	<BANLkTi=Uk+bNB8sCZLVb86q-Kjfx1udEZA@mail.gmail.com>
	<1306518628-sup-5396@brick>
Date: Fri, 27 May 2011 15:29:24 -0400
Message-ID: <BANLkTi=cZ50h50xf_OigTyjdfY_y34AX_g@mail.gmail.com>
Subject: Re: one-time-iterators
From: Austin Clements <amdragon@mit.edu>
To: Patrick Totzke <patricktotzke@googlemail.com>,
	Sebastian Spaeth <sebastian@sspaeth.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: notmuch <notmuch@notmuchmail.org>
Precedence: list

On Fri, May 27, 2011 at 2:04 PM, Patrick Totzke
<patricktotzke@googlemail.com> wrote:
> Excerpts from Austin Clements's message of Fri May 27 03:41:44 +0100 2011=
:
>> >> > > Have you tried simply calling list() on your thread
>> >> > > iterator to see how expensive it is? =A0My bet is that it's quite=
 cheap,
>> >> > > both memory-wise and CPU-wise.
>> >> > Funny thing:
>> >> > =A0q=3DDatabase().create_query('*')
>> >> > =A0time tlist =3D list(q.search_threads())
>> >> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some r=
eason
>> >> > the list constructor must read mere than once from the iterator.
>> >> > So this is not an option, but even if it worked, it would show
>> >> > the same behaviour as my above test..
>> >>
>> >> Interesting. =A0Looks like the Threads class implements __len__ and t=
hat
>> >> its implementation exhausts the iterator. =A0Which isn't a great idea=
 in
>> >> itself, but it turns out that Python's implementation of list() calls
>> >> __len__ if it's available (presumably to pre-size the list) before
>> >> iterating over the object, so it exhausts the iterator before even
>> >> using it.
>> >>
>> >> That said, if list(q.search_threads()) did work, it wouldn't give you
>> >> better performance than your experiment above.
> true. Nevertheless I think that list(q.search_threads())
> should be equivalent to [t for t in q.search_threads()], which is
> something to be fixed in the bindings. Should I file an issue somehow?
> Or is enough to state this as a TODO here on the list?

Yes, they should be equivalent.

Sebastian was thinking about fixing the larger issue of generator
exhaustion, which would address this, though the performance would
depend on the cost of iterating twice.  This is why generators
shouldn't support __len__.  Unfortunately, it's probably hard to get
rid of at this point and I doubt there's a way to tell list() to
overlook the presence of a __len__ method.

>> >> > would it be very hard to implement a Query.search_thread_ids() ?
>> >> > This name is a bit off because it had to be done on a lower level.
>> >>
>> >> Lazily fetching the thread metadata on the C side would probably
>> >> address your problem automatically. =A0But what are you doing that
>> >> doesn't require any information about the threads you're manipulating=
?
>> > Agreed. Unfortunately, there seems to be no way to get a list of threa=
d
>> > ids or a reliable iterator thereof by using the current python binding=
s.
>> > It would be enough for me to have the ids because then I could
>> > search for the few threads I actually need individually on demand.
>>
>> There's no way to do that from the C API either, so don't feel left
>> out. =A0]:--8) =A0It seems to me that the right solution to your problem
>> is to make thread information lazy (effectively, everything gathered
>> in lib/thread.cc:_thread_add_message). =A0Then you could probably
>> materialize that iterator cheaply.
> Alright. I'll put this on my mental notmuch wish list and
> hope that someone will have addressed this before I run out of
> ideas how to improve my UI and have time to look at this myself.
> For now, I go with the [t.get_thread_id for t in q.search_threads()]
> approach to cache the thread ids myself and live with the fact that
> this takes time for large result sets.
>
>> In fact, it's probably worth
>> trying a hack where you put dummy information in the thread object
>> from _thread_add_message and see how long it takes just to walk the
>> iterator (unfortunately I don't think profiling will help much here
>> because much of your time is probably spent waiting for I/O).
> I don't think I understand what you mean by dummy info in a thread
> object.

In _thread_add_message, rather than looking up the message's author,
subject, etc, just hard-code some dummy values.  Performance-wise,
this would simulate making the thread metadata lookup lazy, so you
could see if making this lazy would address your problem.