From: Mark Walters <markwalters1009@gmail.com>
To: Austin Clements <amdragon@MIT.EDU>
Subject: Re: [PATCH] Automatically exclude tags in notmuch-show
In-Reply-To: <20120122181609.GQ16740@mit.edu>
References: <874nvric7c.fsf@qmul.ac.uk>
	<1327010583-23954-1-git-send-email-markwalters1009@gmail.com>
	<20120119225910.GT16740@mit.edu> <871uqvgrnm.fsf@qmul.ac.uk>
	<20120120171801.GA16740@mit.edu> <20120122181609.GQ16740@mit.edu>
User-Agent: Notmuch/0.11+99~g7f60f7e (http://notmuchmail.org) Emacs/23.2.1
	(i486-pc-linux-gnu)
Date: Mon, 23 Jan 2012 01:13:29 +0000
Message-ID: <87zkdfgr0m.fsf@qmul.ac.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: notmuch@notmuchmail.org
Precedence: list

On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth myself on Jan 20 at 12:18 pm:
> > Quoth Mark Walters on Jan 20 at 12:10 am:
> > > 
> > > Ok Having said this is trivial I have found a problem. What should
> > > notmuch do if you do something like
> > > 
> > > notmuch show id:<some-id>
> > > and that message is marked with a deleted tag? To be consistent with the
> > > other cases (where a deleted message is in a matched thread) we might
> > > want to return the message with the not-matched flag set (eg in
> > > JSON). But my patch doesn't, as it never even sees the thread since it
> > > doesn't match.
> > > 
> > > Looking at notmuch-show.c I think we should not apply the exclude tags
> > > to do_show_single, but usually should apply it to do_show. One solution
> > > which is simple and is at least close to right would be to get do_show
> > > to return the number of threads found. If this is zero then retry the
> > > query without the excludes (possible setting the match_flag to zero on
> > > each message since we know it does not match)
> > > 
> > > This is not a completely correct solution as if you ask notmuch-show to
> > > show more than one thread it might  threads which only contain deleted
> > > messages.
> > > 
> > > I can't see other good possibilities without slowing down the normal
> > > path a lot (eg find all threads that match the original query and then
> > > apply the argument above).
> > > 
> > > Any thoughts?
> > 
> > Oh dear.
> > 
> > Well, here's one idea.  Instead of doing a single thread query in
> > show, do a thread query without the exclusions and then a message
> > query with the exclusions.  Output all of the messages from the first
> > query, but use the results of the second query to determine which
> > messages are "matched".  The same could be accomplished in the library
> > somewhat more efficiently, but it's not obvious to me what the API
> > would be.
> 
> Here's a slightly crazier idea that's more library-invasive than the
> original approach, but probably better in the long run.
> 
> Have notmuch_query_search_* return everything and make exclusion a
> message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
> "matched" to mean "matched and not excluded" (specifically, a message
> would have the match flag or the excluded flag or neither, but not
> both).  Search would skip threads with zero matched messages and I
> think show would Just Work.
> 
> I can think of two ways to implement this.  notmuch_query_search_*
> could perform both the original query and the query with exclusions
> and use the docid set from the second to compute the "excluded"
> message flag.  Alternatively, it could examine the tags of each
> message directly to compute the flag.  The latter is probably easier
> to implement, but probably slower.
> 
> Thoughts?

I have now thought about this some more and think I understand your idea
(and how it would work) rather better now. 

I would suggest one small change: the flags for the messages returned
should be "independent": so a message can match the query or not, and it
can be excluded or not, with all 4 combinations being possible. (The
consumer of notmuch_query_search_* would extract the information it
wanted.)

I have thought about some implementation ideas but I think sorting is
going to be the deciding factor: what order should
notmuch_query_search_* return messages/threads? 

For notmuch_query_search_messages either it returns them all together
with the excluded messages marked, or returns all included ones, and
then all excluded one.

For notmuch_query_search_threads it is less clear. Currently it returns
threads in order of first matching message. It is not clear what
matching means now: is matching and included, or just matching? If the
former then we will be returning some threads with no matching and
included messages so we need to decide where to put them in the order.

If we sort in both cases just on matching then we have the same
output/sort as notmuch pre-excluded flags, just the frontends
notmuch-search/show can decide to omit some lines/results. Note that
after omitting "excluded" lines the thread sort would be different from
the current notmuch-with-excluded implementation.

Whereas if we sort based on matching and included, we keep the current
sort order with some stuff appended.

As regards implementation I think notmuch_query_search_messages is the
crucial place: once that returns one of its two orders the rest sort of
takes care of itself.

Best wishes

Mark