DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma;
	h=cc:subject:from:to:in-reply-to:references:date:message-id
	:user-agent:content-transfer-encoding:mime-version:content-type;
	b=YFD7gqVUyVdVr/z9fxWXhqKiX0VExjnew7/MVXEb9kl6TPGyYbaAQSRNt7KLBv+lYY
	gnro4nTO+OBnxAjJYlttdZNhtZ3HaJ3uw/WRAMOZa/X9Q8kalgQjqIEBr7/pSmuVSU63
	83ys/LCr57pZ/zwYw8sDjE6Pr0pJXpmwST1Ts=
Subject: Re: one-time-iterators
From: Patrick Totzke <patricktotzke@googlemail.com>
To: Austin Clements <amdragon@mit.edu>
In-reply-to: <BANLkTi=Uk+bNB8sCZLVb86q-Kjfx1udEZA@mail.gmail.com>
References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>
	<BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>
	<1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu>
	<1306446621-sup-3184@brick>
	<BANLkTi=Uk+bNB8sCZLVb86q-Kjfx1udEZA@mail.gmail.com>
Date: Fri, 27 May 2011 19:04:27 +0100
Message-Id: <1306518628-sup-5396@brick>
User-Agent: Sup/git
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-1306519467-975049-22881-1746-2-=";
	protocol="application/pgp-signature"
Cc: Patrick Totzke <patricktotzke@googlemail.com>,
	notmuch <notmuch@notmuchmail.org>
Precedence: list


--=-1306519467-975049-22881-1746-2-=
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Excerpts from Austin Clements's message of Fri May 27 03:41:44 +0100 2011=
:
> >> > > Have you tried simply calling list() on your thread
> >> > > iterator to see how expensive it is? =C2=A0My bet is that it's q=
uite cheap,
> >> > > both memory-wise and CPU-wise.
> >> > Funny thing:
> >> > =C2=A0q=3DDatabase().create_query('*')
> >> > =C2=A0time tlist =3D list(q.search_threads())
> >> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some =
reason
> >> > the list constructor must read mere than once from the iterator.
> >> > So this is not an option, but even if it worked, it would show
> >> > the same behaviour as my above test..
> >>
> >> Interesting. =C2=A0Looks like the Threads class implements __len__ a=
nd that
> >> its implementation exhausts the iterator. =C2=A0Which isn't a great =
idea in
> >> itself, but it turns out that Python's implementation of list() call=
s
> >> __len__ if it's available (presumably to pre-size the list) before
> >> iterating over the object, so it exhausts the iterator before even
> >> using it.
> >>
> >> That said, if list(q.search_threads()) did work, it wouldn't give yo=
u
> >> better performance than your experiment above.
true. Nevertheless I think that list(q.search_threads())
should be equivalent to [t for t in q.search_threads()], which is
something to be fixed in the bindings. Should I file an issue somehow?
Or is enough to state this as a TODO here on the list?

> >> > would it be very hard to implement a Query.search_thread_ids() ?
> >> > This name is a bit off because it had to be done on a lower level.=

> >>
> >> Lazily fetching the thread metadata on the C side would probably
> >> address your problem automatically. =C2=A0But what are you doing tha=
t
> >> doesn't require any information about the threads you're manipulatin=
g?
> > Agreed. Unfortunately, there seems to be no way to get a list of thre=
ad
> > ids or a reliable iterator thereof by using the current python bindin=
gs.
> > It would be enough for me to have the ids because then I could
> > search for the few threads I actually need individually on demand.
> =

> There's no way to do that from the C API either, so don't feel left
> out.  ]:--8)  It seems to me that the right solution to your problem
> is to make thread information lazy (effectively, everything gathered
> in lib/thread.cc:_thread_add_message).  Then you could probably
> materialize that iterator cheaply. =

Alright. I'll put this on my mental notmuch wish list and =

hope that someone will have addressed this before I run out of
ideas how to improve my UI and have time to look at this myself.
For now, I go with the [t.get_thread_id for t in q.search_threads()]
approach to cache the thread ids myself and live with the fact that
this takes time for large result sets.

> In fact, it's probably worth
> trying a hack where you put dummy information in the thread object
> from _thread_add_message and see how long it takes just to walk the
> iterator (unfortunately I don't think profiling will help much here
> because much of your time is probably spent waiting for I/O).
I don't think I understand what you mean by dummy info in a thread
object.

> I don't think there would be any downside to doing this for eager
> consumers like the CLI.
one should think so, yes.
/p

--=-1306519467-975049-22881-1746-2-=
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3f56sACgkQlDQDZ9fWxaqA9QCgpvXeNv8MJkKU+4knpY7VJC24
VUwAoK5dFwwmSkEhtD8dLt2EYc9PqQTb
=t6vZ
-----END PGP SIGNATURE-----

--=-1306519467-975049-22881-1746-2-=--