From: Mark Walters <markwalters1009@gmail.com>
To: notmuch@notmuchmail.org
Subject: [PATCH 0/3] Outline fix for emacs tagging race
Date: Fri,  9 Nov 2012 18:58:08 +0000
Message-Id: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com>
Precedence: list

For a long time [1] there have been two related races in tagging from
the search buffer.

The first is that when tagging (including archiving) a thread message
which arrived after the buffer was created may also be tagged. This is
because the tagging is done based on the thread-id not on the
individual messages.

The second is when using the '*' command to tag all messages. This is
not quite the same as this command only tags messages matching the
query not all messages in all threads that contain a message matching
the query. Thus if more messages now match than when the buffer was
created (eg some external tagging script has run) then this command
can unexpectedly tag these messages too.

One solution that was discussed in [2] was for the search output of
notmuch to include the message-ids of both matching and non-matching
messages. At that time that was difficult to implement as it was
unclear how to escape the message ids when using the text
format. Since emacs now uses JSON for search mode this problem is
solved.

This patch series implements the above mentioned solution and seems to
work except for one problem.

Since emacs now tags each message in a thread in the search buffer it
is easy to ask it to tag a lot of messages. This could be done
individually which would be ridiculously slow so instead they are all
done in one batch. But now it is relatively easy to take notmuch over
the threshold for ARG_MAX.

In [3] Tomi did some experiments and found on a standard Debian system
with getconf ARG_MAX =131072 that command lines with 10000 200 byte
arguments worked. I am a little puzzled by that as I get the same
results and I getconf ARG_MAX gives 2097152 for me.

More importantly though, when trying to execute a command from emacs I
am finding that 131072 is the limit on the command length in bytes and
we can hit this with something around 1500 messages (see end for a
very hacky emacs test script). This is probably more than we can
expect in a single thread so tagging from show is probably safe but it
does cause a problem when tagging from search.

I can think of several possible solutions (e.g., batch it in my new
stuff, put some batching in notmuch-tag, all notmuch tag to read a
query from stdin). But before any larger more intrusive change do
people like the general approach? Does anyone have a good way to get
round the command line size problem?

Best wishes 

Mark


[1] id:87ocmtg9ni.fsf@yoom.home.cworth.org
[2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=UN7McECrg@mail.gmail.com
[3] id:m2liody7av.fsf@guru.guru-group.fi

Mark Walters (3):
  test: test for race when tagging from emacs search
  cli: all search mode to include msg-ids with JSON output
  emacs: make emacs use message-ids for tagging

 emacs/notmuch.el |   22 ++++++++++++++++++++--
 notmuch-search.c |   40 ++++++++++++++++++++++++++++++++++++++--
 test/emacs       |   21 +++++++++++++++++++++
 3 files changed, 79 insertions(+), 4 deletions(-)


TEST SCRIPT

(progn
  (setq bigstring "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
  (setq n 1310)
  (setq big nil)
  (while (> n 0)
    (setq n (1- n))
    (setq big (concat big (format "%s" n) " " bigstring)))
  (call-process "echo" nil t nil big))



-- 
1.7.9.1