From 1de885ec3fac90c43cabc6ffd0e62b654022f325 Mon Sep 17 00:00:00 2001 From: Austin Clements Date: Sun, 11 Nov 2012 00:29:47 +1900 Subject: [PATCH] Re: [PATCH 0/3] Outline fix for emacs tagging race --- 71/04c7e75975c7dd2b2a78e01ba28d078436f0d1 | 156 ++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 71/04c7e75975c7dd2b2a78e01ba28d078436f0d1 diff --git a/71/04c7e75975c7dd2b2a78e01ba28d078436f0d1 b/71/04c7e75975c7dd2b2a78e01ba28d078436f0d1 new file mode 100644 index 000000000..d146489e9 --- /dev/null +++ b/71/04c7e75975c7dd2b2a78e01ba28d078436f0d1 @@ -0,0 +1,156 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 2536E431FB6 + for ; Fri, 9 Nov 2012 21:29:53 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id l7LoA5VohaN4 for ; + Fri, 9 Nov 2012 21:29:51 -0800 (PST) +Received: from dmz-mailsec-scanner-5.mit.edu (DMZ-MAILSEC-SCANNER-5.MIT.EDU + [18.7.68.34]) + by olra.theworths.org (Postfix) with ESMTP id 03E3F431FAE + for ; Fri, 9 Nov 2012 21:29:50 -0800 (PST) +X-AuditID: 12074422-b7f746d0000008cc-62-509de64e33f1 +Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) + by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP + id 9F.33.02252.E46ED905; Sat, 10 Nov 2012 00:29:50 -0500 (EST) +Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) + by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id qAA5TnpI014822; + Sat, 10 Nov 2012 00:29:50 -0500 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id qAA5Tl59008088 + (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); + Sat, 10 Nov 2012 00:29:49 -0500 (EST) +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77) + (envelope-from ) + id 1TX3db-0000pf-J4; Sat, 10 Nov 2012 00:29:47 -0500 +Date: Sat, 10 Nov 2012 00:29:47 -0500 +From: Austin Clements +To: Mark Walters +Subject: Re: [PATCH 0/3] Outline fix for emacs tagging race +Message-ID: <20121110052947.GK22284@mit.edu> +References: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFupnleLIzCtJLcpLzFFi42IR4hRV1vV7NjfAYMsMcYvVc3ksrt+cyezA + 5LFz1l12j2erbjEHMEVx2aSk5mSWpRbp2yVwZew5f4W14LB8xa+bKxkbGGdLdjFyckgImEjs + mbOeEcIWk7hwbz1bFyMXh5DAPkaJzq3XGSGcDYwSR37/ZoZwTjJJdF/fyQrhLGGUmP1oKVg/ + i4CqxOYdb8BsNgENiW37l4PZIgI6ErcPLWAHsZkFpCW+/W5mArGFBawl3t79D1bDC1Qzd9lx + ZhBbSMBTYk3TCzaIuKDEyZlPWCB6tSRu/HsJ1MsBNmf5Pw6QMKeAl8TL06/AykUFVCSmnNzG + NoFRaBaS7llIumchdC9gZF7FKJuSW6Wbm5iZU5yarFucnJiXl1qka6qXm1mil5pSuokRFNbs + Lko7GH8eVDrEKMDBqMTDmxA+N0CINbGsuDL3EKMkB5OSKO/PJ0AhvqT8lMqMxOKM+KLSnNTi + Q4wSHMxKIryv+oFyvCmJlVWpRfkwKWkOFiVx3mspN/2FBNITS1KzU1MLUotgsjIcHEoSvIVP + gRoFi1LTUyvSMnNKENJMHJwgw3mAhseA1PAWFyTmFmemQ+RPMepyHH0z9yGjEEtefl6qlDiv + D0iRAEhRRmke3BxYOnrFKA70ljBvIkgVDzCVwU16BbSECWhJ45E5IEtKEhFSUg2MRZ1vb+yp + +v2mgV1LtbT/tNq+bbmuOttDFevMa2Z+lFe+Wfl4xopNXkozVQ6EZ78KdpRbc7Ihw7pIZ29p + 0mrd0yeW/sufa7I3aEfMUZlHp37zHucNsr97e86FUKcZO1xtOP18Shu877Wqu8gzaJveesnl + uXm26+FQuWs8zDc4VjO+NtMRM9iixFKckWioxVxUnAgAATH5siIDAAA= +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sat, 10 Nov 2012 05:29:53 -0000 + +Quoth Mark Walters on Nov 09 at 6:58 pm: +> For a long time [1] there have been two related races in tagging from +> the search buffer. +> +> The first is that when tagging (including archiving) a thread message +> which arrived after the buffer was created may also be tagged. This is +> because the tagging is done based on the thread-id not on the +> individual messages. +> +> The second is when using the '*' command to tag all messages. This is +> not quite the same as this command only tags messages matching the +> query not all messages in all threads that contain a message matching +> the query. Thus if more messages now match than when the buffer was +> created (eg some external tagging script has run) then this command +> can unexpectedly tag these messages too. +> +> One solution that was discussed in [2] was for the search output of +> notmuch to include the message-ids of both matching and non-matching +> messages. At that time that was difficult to implement as it was +> unclear how to escape the message ids when using the text +> format. Since emacs now uses JSON for search mode this problem is +> solved. +> +> This patch series implements the above mentioned solution and seems to +> work except for one problem. +> +> Since emacs now tags each message in a thread in the search buffer it +> is easy to ask it to tag a lot of messages. This could be done +> individually which would be ridiculously slow so instead they are all +> done in one batch. But now it is relatively easy to take notmuch over +> the threshold for ARG_MAX. +> +> In [3] Tomi did some experiments and found on a standard Debian system +> with getconf ARG_MAX =131072 that command lines with 10000 200 byte +> arguments worked. I am a little puzzled by that as I get the same +> results and I getconf ARG_MAX gives 2097152 for me. +> +> More importantly though, when trying to execute a command from emacs I +> am finding that 131072 is the limit on the command length in bytes and +> we can hit this with something around 1500 messages (see end for a +> very hacky emacs test script). This is probably more than we can +> expect in a single thread so tagging from show is probably safe but it +> does cause a problem when tagging from search. +> +> I can think of several possible solutions (e.g., batch it in my new +> stuff, put some batching in notmuch-tag, all notmuch tag to read a +> query from stdin). But before any larger more intrusive change do +> people like the general approach? Does anyone have a good way to get +> round the command line size problem? +> +> Best wishes +> +> Mark +> +> +> [1] id:87ocmtg9ni.fsf@yoom.home.cworth.org +> [2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=UN7McECrg@mail.gmail.com +> [3] id:m2liody7av.fsf@guru.guru-group.fi + +I'm glad to see someone picking up this bug. Besides somehow dealing +with long command-lines, when we were last exploring this race, I had +found that it was 3-4x more efficient to use Xapian document IDs +directly rather than message IDs [1]. It's probably best *not* to do +this initially for the sake of simplicity, but I think a simple tweak +to your approach would let us seamlessly transition to this in the +future. Rather than extending the JSON output with what are +explicitly message IDs, instead extend the output with opaque queries +that are guaranteed to match the matching/non-matching messages in the +thread and are guaranteed to be combinable, but aren't guaranteed to +be of any particular form. For now, the CLI can simply output id: +queries for these, but in the future we could easily add a special +query syntax for docid queries or, if we ever move to a custom query +parser, support docids in any query. + +For the long command line problem, one easy solution is to support, +say, '-' as a query syntax that means "read the query from stdin." +This would be a simple addition to query_string_from_args and would +work across the CLI. + +[1] id:CAH-f9WsPj=1Eu=g3sOePJgCTBFs6HrLdLq18xMEnJ8aZ00yCEg@mail.gmail.com -- 2.26.2