From: Mark Walters Date: Fri, 9 Nov 2012 18:58:08 +0000 (+0000) Subject: [PATCH 0/3] Outline fix for emacs tagging race X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=c9d8546472aff6492d6dcbd8701baa75cbb008e7;p=notmuch-archives.git [PATCH 0/3] Outline fix for emacs tagging race --- diff --git a/d2/5baad321e5efaf6d14ec4454fad334116d82eb b/d2/5baad321e5efaf6d14ec4454fad334116d82eb new file mode 100644 index 000000000..45c424e63 --- /dev/null +++ b/d2/5baad321e5efaf6d14ec4454fad334116d82eb @@ -0,0 +1,145 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 678C0431FB6 + for ; Fri, 9 Nov 2012 10:58:27 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0.201 +X-Spam-Level: +X-Spam-Status: No, score=0.201 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + FREEMAIL_ENVFROM_END_DIGIT=1, FREEMAIL_FROM=0.001, + RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id pp1zC4URjimz for ; + Fri, 9 Nov 2012 10:58:25 -0800 (PST) +Received: from mail-wg0-f45.google.com (mail-wg0-f45.google.com + [74.125.82.45]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client + certificate requested) by olra.theworths.org (Postfix) with ESMTPS id + 34FC2431FAE for ; Fri, 9 Nov 2012 10:58:25 -0800 + (PST) +Received: by mail-wg0-f45.google.com with SMTP id dq12so2288423wgb.2 + for ; Fri, 09 Nov 2012 10:58:23 -0800 (PST) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; + h=from:to:cc:subject:date:message-id:x-mailer; + bh=iVqVh7xUHX1VPSG0tXROjHgC34l3GXEyAxVmiyl7UI0=; + b=Ak3HOehNKJA4AqbhdaDgL2jJXzmlmY2BcR9rJT3FaFviPiWvvZuP682qIxK95qn6CG + UTX29l7j1NhYK3vEbn2/Pq/rISpq2y/uFas3qhllSf1AoIs/v1Nejsvh0JOZ+/CwW4gx + ZN49TadE0ISl8JOfs4n1jxbBdxom3TSEOpJKUwPmOIlikGBRG9ber0I6uh41CTuDXb+l + Nm8/sCg6vWjnffYSs3eAL5OCxKnenoXJsaS2RItEJI206B1XiPqtuKeArOcpMjlOcyDR + A7gesBy05x79a8fafBJlr+XiIpjBBfO3Lp9p7OpRT/CJmh0M5ZTPKKjgaxK6Dd1jqYRX + IUhQ== +Received: by 10.180.95.201 with SMTP id dm9mr4184109wib.3.1352487502655; + Fri, 09 Nov 2012 10:58:22 -0800 (PST) +Received: from localhost (93-97-24-31.zone5.bethere.co.uk. [93.97.24.31]) + by mx.google.com with ESMTPS id r10sm3779351wiz.0.2012.11.09.10.58.20 + (version=TLSv1/SSLv3 cipher=OTHER); + Fri, 09 Nov 2012 10:58:21 -0800 (PST) +From: Mark Walters +To: notmuch@notmuchmail.org +Subject: [PATCH 0/3] Outline fix for emacs tagging race +Date: Fri, 9 Nov 2012 18:58:08 +0000 +Message-Id: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com> +X-Mailer: git-send-email 1.7.9.1 +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Fri, 09 Nov 2012 18:58:27 -0000 + +For a long time [1] there have been two related races in tagging from +the search buffer. + +The first is that when tagging (including archiving) a thread message +which arrived after the buffer was created may also be tagged. This is +because the tagging is done based on the thread-id not on the +individual messages. + +The second is when using the '*' command to tag all messages. This is +not quite the same as this command only tags messages matching the +query not all messages in all threads that contain a message matching +the query. Thus if more messages now match than when the buffer was +created (eg some external tagging script has run) then this command +can unexpectedly tag these messages too. + +One solution that was discussed in [2] was for the search output of +notmuch to include the message-ids of both matching and non-matching +messages. At that time that was difficult to implement as it was +unclear how to escape the message ids when using the text +format. Since emacs now uses JSON for search mode this problem is +solved. + +This patch series implements the above mentioned solution and seems to +work except for one problem. + +Since emacs now tags each message in a thread in the search buffer it +is easy to ask it to tag a lot of messages. This could be done +individually which would be ridiculously slow so instead they are all +done in one batch. But now it is relatively easy to take notmuch over +the threshold for ARG_MAX. + +In [3] Tomi did some experiments and found on a standard Debian system +with getconf ARG_MAX =131072 that command lines with 10000 200 byte +arguments worked. I am a little puzzled by that as I get the same +results and I getconf ARG_MAX gives 2097152 for me. + +More importantly though, when trying to execute a command from emacs I +am finding that 131072 is the limit on the command length in bytes and +we can hit this with something around 1500 messages (see end for a +very hacky emacs test script). This is probably more than we can +expect in a single thread so tagging from show is probably safe but it +does cause a problem when tagging from search. + +I can think of several possible solutions (e.g., batch it in my new +stuff, put some batching in notmuch-tag, all notmuch tag to read a +query from stdin). But before any larger more intrusive change do +people like the general approach? Does anyone have a good way to get +round the command line size problem? + +Best wishes + +Mark + + +[1] id:87ocmtg9ni.fsf@yoom.home.cworth.org +[2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=UN7McECrg@mail.gmail.com +[3] id:m2liody7av.fsf@guru.guru-group.fi + +Mark Walters (3): + test: test for race when tagging from emacs search + cli: all search mode to include msg-ids with JSON output + emacs: make emacs use message-ids for tagging + + emacs/notmuch.el | 22 ++++++++++++++++++++-- + notmuch-search.c | 40 ++++++++++++++++++++++++++++++++++++++-- + test/emacs | 21 +++++++++++++++++++++ + 3 files changed, 79 insertions(+), 4 deletions(-) + + +TEST SCRIPT + +(progn + (setq bigstring "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") + (setq n 1310) + (setq big nil) + (while (> n 0) + (setq n (1- n)) + (setq big (concat big (format "%s" n) " " bigstring))) + (call-process "echo" nil t nil big)) + + + +-- +1.7.9.1 +