From: Jameson Graef Rollins <jrollins@finestructure.net>
To: Mark Walters <markwalters1009@gmail.com>, notmuch@notmuchmail.org
Subject: Re: Notmuch Pick
In-Reply-To: <87mx3zok3d.fsf@qmul.ac.uk>
References: <87395ump0d.fsf@qmul.ac.uk>
	<87zk80ilt9.fsf@servo.finestructure.net>
	<87mx3zok3d.fsf@qmul.ac.uk>
User-Agent: Notmuch/0.13.2+53~g1567997 (http://notmuchmail.org) Emacs/23.4.1
	(x86_64-pc-linux-gnu)
Date: Tue, 19 Jun 2012 11:52:01 -0700
Message-ID: <87zk7z2kz2.fsf@servo.finestructure.net>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha256; protocol="application/pgp-signature"
Precedence: list

--=-=-=
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 19 2012, Mark Walters <markwalters1009@gmail.com> wrote:
> As you say several tests do need fixing: I
> thought I would leave that until people said they were basically happy
> with the change.=20

Yeah, but failing tests are basically a non-starter for me.  You can
maybe get away with not adding new tests during development, but I'm
unlikely to even try out the patches if a bunch of the existing tests
are failing.

> I am not sure about sanity tests for pick: how would that work while it
> lives in contrib? Obviously it would need some tests before coming in to
> proper mainline.

Since you're not actually submitting these patches yet, does it really
need to live in contrib?  The goal is for it to ultimately *not* live in
contrib, so I would say just go ahead and make them apply in the main
source.

An example of a sanity test I'm talking about is simply display the pick
thread for something and test that it looks like expected.  Sort of a
zeroth order thing.  There are other emacs tests that do the same sort
of thing so it shouldn't be too hard to add.

>> Would it also be useful to make this same change for the search out, for
>> consistency?  I notice the search output now uses newlines between all
>> fields, which should help for asynchronous processing, but it might be
>> nice to put newline separators between the initial and final brackets as
>> well.
>
> Right.=20

I would say go ahead and send to the list any patches that come up in
development that are generally useful.  Especially if they're small like
this they are likely to applied before this series, so it will be
smaller and easier to read when the times comes.

>> df97df62b70b884a1cd367360ed6ff7eda0e8af6
>>     cli: add --headers_only option to notmuch-show.c
>>
>> Your comment in this patch is very interesting:
>>
>>     This is used by notmuch-pick.el (although not needed) because it giv=
es a
>>     speed-up of at least a factor of a two; moreover it reduces the memo=
ry
>>     usage in emacs hugely.
>>
>> The only difference between the regular show json output and the
>> --headers-only output, as far as I can tell, is the presence of the
>> content of text/plain parts if they exist in the message.  We previously
>> had a discussion about the show output not including any part bodies at
>> all, but we decided that the inclusion of text/plain bodies shouldn't
>> affect anything, so why not include them.  If they actually do, then I
>> argue we should just move to having show json not include any body parts
>> at all by default, and just have them be retrieved individually like we
>> do currently for non-text/plain parts.  This would make things cleaner,
>> and would get rid of the need to have this extra option, which really
>> doesn't produce a significantly different output.
>
> For one use of pick (displaying the structure of a single thread) this
> is not important (and the asynchronous stuff is irrelevant too). For
> another use of displaying the thread structure of a whole "folder" of
> messages it is important. For example the output of=20
>
> notmuch show tag:notmuch
>
> is about 70MB on my system, whereas with the --headers-only option this
> drops to about 7MB. Note Emacs uses substantially  more than this much
> memory to actually process the JSON, and on a low-powered laptop this
> is enough to cause a swap-storm.
>
> Your suggestion of just not including the text/plain is nice for
> notmuch-pick, and is very simple (a single line change). However, it
> does seem to slow the emacs show mode down noticeably on large threads
> (something like 2s to 4s on a thread with 180 messages) so I worry that
> this change might annoy people. What do you think?

Threads that long are already a bit of a pain to deal with.  But I think
this reveals a weakness in the way we're displaying threads more than
anything.  It seems silly to me that we would retrieve all 180 messages
From=20such a thread, and format all of them, only to then hide all but
the one that the reader is seeing.  I think this "pick" series
illustrates this nicely.  Maybe it would be better to refactor the
current emacs show mode to only retrieve parts for messages that are
being displayed.  I'm not sure how difficult that would be, though.

I would also like to have access to more of the message headers in show
mode.  For instance, in emacs I want to access more headers with
notmuch-message-headers than are currently available.  Having json
always include all headers might be a bit of a performance hit for some
messages with lots of headers, but it would allow callers access to
headers they don't currently have access to at all.

So I think it would be nice and clean if the default json output
included all the message headers and the message structure, and then
part contents could be retrieved individually.  I imagine show mode
could be restructured such that it could use this efficiently.

jamie.

PS. where did the name "pick" come from?  It doesn't seem to fit with
the functionality to me.

--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJP4MpSAAoJEO00zqvie6q8HWIP/3zuS2RWqZVQyvq7SPhwCtFB
9zmWoYvHtTwmeGqgBCAbfVKZVzczSSVW7JIltDL5hT91IDnBY5VT5vifz0fecBz+
1U9H0DT5fdU84WB8kzGgvzptHVEbO0QMWodiPRbFjfBk1WovB7EVN1OPfBQE27lw
x7okLDEKg0wAXLdFnXC1swoPCxSYVv2Zr+joJwLBmyLBW85Hm+cEmP/ZrMAVrmBe
zbl9phzD7XbSudU5UE0sg1AjKfcYbZpeqmgKqdXn38e+0JHQCZsWptBfu/8SDz09
9A6+dYSIVKJ8Gv25JkdI9gIKHgh8exc05DH+8KgDEa+L4srpbHpG3DfvedS1WpBy
n4xQchF1DXcTk89G228pP0EaNLHR6eF82TqJ2vIfe+csTU0VnaxFsTbXS6Skup1F
UvTedyC005el3Ej7quLI43JQa39CktsVT8sEUiqdi9HQpcdmC/PiU/hHdloLjEvS
ym1qy45Vfi/hKmoLKVCERmkMVn+i0lkW76m7FVX1oDdM5LjqIYWRDF2JdhP5/whW
VT2YCDbwdejKMbB906fZk9tMpvi/ui477gCOYMwf056kJ0iftsj/Jxolc08aHTs5
uyRd+5XWdUjCMdGb6NuuxSpoG1B7k7M5iZmSBaKmyDKR+MMPuGUzHBykbehTU4EF
R3hVLYoSH5if5SkBFJv5
=5XeH
-----END PGP SIGNATURE-----
--=-=-=--