DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma;
	h=cc:subject:from:to:in-reply-to:references:date:message-id
	:user-agent:content-transfer-encoding:mime-version:content-type;
	b=gaY30vbWZzhcQyPrWfk4e0YeiCbdUoIUALBk0LGRGyMQfcw+TSu0NBm3Yxhh5gddLv
	SsEW0bLpC/sPUaPECheXRG082EEyuGMqyWoLgOW4ibwP8I95RLAC7WbjFZdDoVzDlI3V
	o0XcRMPlFw7SzQwl84Tudhn/Bew6xQxXzNYfk=
Subject: Re: one-time-iterators
From: Patrick Totzke <patricktotzke@googlemail.com>
To: Austin Clements <amdragon@mit.edu>
In-reply-to: <20110526214302.GR29861@mit.edu>
References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>
	<BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>
	<1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu>
Date: Thu, 26 May 2011 23:22:11 +0100
Message-Id: <1306446621-sup-3184@brick>
User-Agent: Sup/git
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-1306448531-827817-32122-6706-2-=";
	protocol="application/pgp-signature"
Cc: notmuch@notmuchmail.org
Precedence: list


--=-1306448531-827817-32122-6706-2-=
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Excerpts from Austin Clements's message of Thu May 26 22:43:02 +0100 2011=
:
> http://notmuch.198994.n3.nabble.com/notmuch-s-idea-of-concurrency-faili=
ng-an-invocation-tp2373468p2565731.html
ah, good old peterson :P thanks.

> > > Though, Patrick, that solution doesn't address your problem.=C2=A0 =
On the
> > > other hand, it's not clear to me what concurrent access semantics
> > > you're actually expecting.=C2=A0 I suspect you don't want the remai=
ning
> > > iteration to reflect the changes, since your changes could equally
> > > well have affected earlier iteration results.=C2=A0
> > That's right. =

> > > But if you want a
> > > consistent view of your query results, something's going to have to=

> > > materialize that iterator, and it might as well be you (or Xapian
> > > would need more sophisticated concurrency control than it has).=C2=A0=
 But
> > > this shouldn't be expensive because all you need to materialize are=

> > > the document ids; you shouldn't need to eagerly fetch the per-threa=
d
> > > information.  =

> > I thought so, but it seems that Query.search_threads() already
> > caches more than the id of each item. Which is as expected
> > because it is designed to return thread objects, not their ids.
> > As you can see above, this _is_ too expensive for me.
> =

> I'd forgotten that constructing threads on the C side was eager about
> the thread tags, author list and subject (which, without Istvan's
> proposed patch, even requires opening and parsing the message file).
> This is probably what's killing you.
> =

> Out of curiosity, what is your situation that you won't wind up paying
> the cost of this iteration one way or the other and that the latency
> of doing these tag changes matters?

I'm trying to implement a terminal interface for notmuch in python
that resembles sup.
For the search results view, i read an initial portion from a Threads ite=
rator =

to fill my teminal window with threadline-widgets. Obviously, for a
large number of results I don't want to go through all of them.
The problem arises if you toggle a tag on the selected threadline and aft=
erwards
continue to scroll down.

> > > Have you tried simply calling list() on your thread
> > > iterator to see how expensive it is?  My bet is that it's quite che=
ap,
> > > both memory-wise and CPU-wise.
> > Funny thing:
> >  q=3DDatabase().create_query('*')
> >  time tlist =3D list(q.search_threads())
> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some rea=
son
> > the list constructor must read mere than once from the iterator.
> > So this is not an option, but even if it worked, it would show
> > the same behaviour as my above test..
> =

> Interesting.  Looks like the Threads class implements __len__ and that
> its implementation exhausts the iterator.  Which isn't a great idea in
> itself, but it turns out that Python's implementation of list() calls
> __len__ if it's available (presumably to pre-size the list) before
> iterating over the object, so it exhausts the iterator before even
> using it.
> =

> That said, if list(q.search_threads()) did work, it wouldn't give you
> better performance than your experiment above.
> =

> > would it be very hard to implement a Query.search_thread_ids() ?
> > This name is a bit off because it had to be done on a lower level.
> =

> Lazily fetching the thread metadata on the C side would probably
> address your problem automatically.  But what are you doing that
> doesn't require any information about the threads you're manipulating?
Agreed. Unfortunately, there seems to be no way to get a list of thread
ids or a reliable iterator thereof by using the current python bindings.
It would be enough for me to have the ids because then I could
search for the few threads I actually need individually on demand.

Here is the branch in which I'm trying out these things. Sorry for the
messy code, its late :P
https://github.com/pazz/notmuch-gui/tree/toggletags

/p

--=-1306448531-827817-32122-6706-2-=
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3e0pMACgkQlDQDZ9fWxaqKuQCfTm9nYgX4HJb2+tZlZUuGnXfp
mBcAn1pyXaVzNQR9dJfIgHCyGSv/G6hl
=X10m
-----END PGP SIGNATURE-----

--=-1306448531-827817-32122-6706-2-=--