From: Mark Walters <markwalters1009@gmail.com>
To: Jameson Graef Rollins <jrollins@finestructure.net>,
 notmuch@notmuchmail.org
Subject: Re: Notmuch Pick
In-Reply-To: <87zk80ilt9.fsf@servo.finestructure.net>
References: <87395ump0d.fsf@qmul.ac.uk>
	<87zk80ilt9.fsf@servo.finestructure.net>
User-Agent: Notmuch/0.13.2+63~g548a9bf (http://notmuchmail.org) Emacs/23.4.1
	(x86_64-pc-linux-gnu)
Date: Tue, 19 Jun 2012 08:07:50 +0100
Message-ID: <87mx3zok3d.fsf@qmul.ac.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Precedence: list


On Mon, 18 Jun 2012, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Sat, Jun 16 2012, Mark Walters <markwalters1009@gmail.com> wrote:
>> Since I have had various requests for notmuch pick
>> (id:"1329096015-8078-2-git-send-email-markwalters1009@gmail.com") so I
>> have started a git repository at
>> git://github.com/markwalters1009/notmuch.git
>>
>> The branch pick-6 is the current version. My intention is to start a new
>> branch each time I rebase on to current master but this may change. (I
>> suggest that people do not rely on a consistent history for this repository)
>
> Hey, Mark.  I took a look at this and I have a couple of comments:
>
> 726e11aff69e10499a9855e0ac2f15e518985c1f
>     cli: notmuch-show with framing newlines between threads in JSON.
>
> This patch introduces a change in the json output format, but there is
> no subsequent update of the test suite, so it's causing a lot of test
> failures.  Obviously this needs to be fixed, but it would probably be
> nice to include a couple of other tests for the pick output itself.  At
> the very least a sanity test to check that it's working at all would be
> sufficient initially.

Hi 

As you say several tests do need fixing: I
thought I would leave that until people said they were basically happy
with the change. 

I am not sure about sanity tests for pick: how would that work while it
lives in contrib? Obviously it would need some tests before coming in to
proper mainline.

> Would it also be useful to make this same change for the search out, for
> consistency?  I notice the search output now uses newlines between all
> fields, which should help for asynchronous processing, but it might be
> nice to put newline separators between the initial and final brackets as
> well.

Right. 

> df97df62b70b884a1cd367360ed6ff7eda0e8af6
>     cli: add --headers_only option to notmuch-show.c
>
> Your comment in this patch is very interesting:
>
>     This is used by notmuch-pick.el (although not needed) because it gives a
>     speed-up of at least a factor of a two; moreover it reduces the memory
>     usage in emacs hugely.
>
> The only difference between the regular show json output and the
> --headers-only output, as far as I can tell, is the presence of the
> content of text/plain parts if they exist in the message.  We previously
> had a discussion about the show output not including any part bodies at
> all, but we decided that the inclusion of text/plain bodies shouldn't
> affect anything, so why not include them.  If they actually do, then I
> argue we should just move to having show json not include any body parts
> at all by default, and just have them be retrieved individually like we
> do currently for non-text/plain parts.  This would make things cleaner,
> and would get rid of the need to have this extra option, which really
> doesn't produce a significantly different output.

For one use of pick (displaying the structure of a single thread) this
is not important (and the asynchronous stuff is irrelevant too). For
another use of displaying the thread structure of a whole "folder" of
messages it is important. For example the output of 

notmuch show tag:notmuch

is about 70MB on my system, whereas with the --headers-only option this
drops to about 7MB. Note Emacs uses substantially  more than this much
memory to actually process the JSON, and on a low-powered laptop this
is enough to cause a swap-storm.

Your suggestion of just not including the text/plain is nice for
notmuch-pick, and is very simple (a single line change). However, it
does seem to slow the emacs show mode down noticeably on large threads
(something like 2s to 4s on a thread with 180 messages) so I worry that
this change might annoy people. What do you think?

Many thanks for the comments!

Mark