DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=sender:from:to:subject:in-reply-to:references:date:message-id
	:mime-version:content-type;
	b=O1pO1W0HnO+q0JJJ+APkG54Ry3k50hwTzm97kTEL/SZUKlZKms4/uuXjxsEg8rq3I1
	V8RPGQNSzISb0H5k8x9izxeHa9wphzqrMEAbTXsz1H3oaIsd8eWvTgPBjRs3GRIkw02l
	VlOTLR0/4StyKg6JIeDpq8NpLP1lMKIz0z93Q=
Sender: Jed Brown <five9a2@gmail.com>
From: Jed Brown <jed@59A2.org>
To: Keith Packard <keithp@keithp.com>, Carl Worth <cworth@cworth.org>,
	notmuch@notmuchmail.org
In-Reply-To: <yuniqd0oae9.fsf@aiko.keithp.com>
References: <yun7htkcou6.fsf@aiko.keithp.com>
	<87639122sx.fsf@yoom.home.cworth.org> <yunhbslpv3q.fsf@aiko.keithp.com>
	<87bpismx21.fsf@yoom.home.cworth.org> <yuniqd0oae9.fsf@aiko.keithp.com>
Date: Tue, 24 Nov 2009 11:20:02 +0100
Message-ID: <87tywkp6lp.fsf@59A2.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Subject: Re: [notmuch] notmuch 'index' mode.
Precedence: list

On Mon, 23 Nov 2009 19:43:26 -0800, Keith Packard <keithp@keithp.com> wrote:
> On Mon, 23 Nov 2009 19:16:54 -0800, Carl Worth <cworth@cworth.org> wrote:
> > Then tags become something that are just for manual manipulation. What
> > do you think?

I really think this is the way to go.

> And disadvantages as searching might actually be slow at some point?

If this happens, there is nothing to prevent notmuch from caching the
search by actually writing a corresponding tag.  This could be made
automatic by logging the cost of each named search (and perhaps the
frequency of making that search), and using a tradeoff function to
decide which searches to optimize.  Once a search was selected for
optimization, there are at least two alternatives, depending on which
queries xapian can answer quickly.

  1. Log the time and spawn an asynchronous notmuch tag process.  Searches
  for
  
    vtag:named-search
  
  (vtag: doesn't need to be a keyword, but I'm only distinguishing here
  for clarity) which was normally translated into
  
     long search expression
  
  becomes
  
    tag:named-search OR newer:timestamp AND (long search expression)
  
  This option guarantees that notmuch new remains fast and simple because
  it does no special tagging, but this query might not be any better.
  
  2. Inform notmuch new that it should apply the chosen tag to messages
  matching the query and spawn the asynchronous notmuch tag process.  Once
  the tag process has finished, searches for vtag:named-search are
  translated to tag:named-search.  This implies concurrent modification of
  the database, otherwise it would cause a stall in incoming mail (not
  important if mass tagging somehow became faster).
  

Admittedly my archive is not huge (100k messages in 3.5 GB plus 1.1 GB
for .notmuch) but queries returning a reasonable number of messages are
still quite fast.  Additionally, searches for tags do not seem to be
greatly faster than queries for complex queries returning a similar
number of results.

Jed