Date: Sat, 10 Nov 2012 00:29:47 -0500
From: Austin Clements <amdragon@MIT.EDU>
To: Mark Walters <markwalters1009@gmail.com>
Subject: Re: [PATCH 0/3] Outline fix for emacs tagging race
Message-ID: <20121110052947.GK22284@mit.edu>
References: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: notmuch@notmuchmail.org
Precedence: list

Quoth Mark Walters on Nov 09 at  6:58 pm:
> For a long time [1] there have been two related races in tagging from
> the search buffer.
> 
> The first is that when tagging (including archiving) a thread message
> which arrived after the buffer was created may also be tagged. This is
> because the tagging is done based on the thread-id not on the
> individual messages.
> 
> The second is when using the '*' command to tag all messages. This is
> not quite the same as this command only tags messages matching the
> query not all messages in all threads that contain a message matching
> the query. Thus if more messages now match than when the buffer was
> created (eg some external tagging script has run) then this command
> can unexpectedly tag these messages too.
> 
> One solution that was discussed in [2] was for the search output of
> notmuch to include the message-ids of both matching and non-matching
> messages. At that time that was difficult to implement as it was
> unclear how to escape the message ids when using the text
> format. Since emacs now uses JSON for search mode this problem is
> solved.
> 
> This patch series implements the above mentioned solution and seems to
> work except for one problem.
> 
> Since emacs now tags each message in a thread in the search buffer it
> is easy to ask it to tag a lot of messages. This could be done
> individually which would be ridiculously slow so instead they are all
> done in one batch. But now it is relatively easy to take notmuch over
> the threshold for ARG_MAX.
> 
> In [3] Tomi did some experiments and found on a standard Debian system
> with getconf ARG_MAX =131072 that command lines with 10000 200 byte
> arguments worked. I am a little puzzled by that as I get the same
> results and I getconf ARG_MAX gives 2097152 for me.
> 
> More importantly though, when trying to execute a command from emacs I
> am finding that 131072 is the limit on the command length in bytes and
> we can hit this with something around 1500 messages (see end for a
> very hacky emacs test script). This is probably more than we can
> expect in a single thread so tagging from show is probably safe but it
> does cause a problem when tagging from search.
> 
> I can think of several possible solutions (e.g., batch it in my new
> stuff, put some batching in notmuch-tag, all notmuch tag to read a
> query from stdin). But before any larger more intrusive change do
> people like the general approach? Does anyone have a good way to get
> round the command line size problem?
> 
> Best wishes 
> 
> Mark
> 
> 
> [1] id:87ocmtg9ni.fsf@yoom.home.cworth.org
> [2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=UN7McECrg@mail.gmail.com
> [3] id:m2liody7av.fsf@guru.guru-group.fi

I'm glad to see someone picking up this bug.  Besides somehow dealing
with long command-lines, when we were last exploring this race, I had
found that it was 3-4x more efficient to use Xapian document IDs
directly rather than message IDs [1].  It's probably best *not* to do
this initially for the sake of simplicity, but I think a simple tweak
to your approach would let us seamlessly transition to this in the
future.  Rather than extending the JSON output with what are
explicitly message IDs, instead extend the output with opaque queries
that are guaranteed to match the matching/non-matching messages in the
thread and are guaranteed to be combinable, but aren't guaranteed to
be of any particular form.  For now, the CLI can simply output id:
queries for these, but in the future we could easily add a special
query syntax for docid queries or, if we ever move to a custom query
parser, support docids in any query.

For the long command line problem, one easy solution is to support,
say, '-' as a query syntax that means "read the query from stdin."
This would be a simple addition to query_string_from_args and would
work across the CLI.

[1] id:CAH-f9WsPj=1Eu=g3sOePJgCTBFs6HrLdLq18xMEnJ8aZ00yCEg@mail.gmail.com