--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 2536E431FB6\r
+ for <notmuch@notmuchmail.org>; Fri, 9 Nov 2012 21:29:53 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id l7LoA5VohaN4 for <notmuch@notmuchmail.org>;\r
+ Fri, 9 Nov 2012 21:29:51 -0800 (PST)\r
+Received: from dmz-mailsec-scanner-5.mit.edu (DMZ-MAILSEC-SCANNER-5.MIT.EDU\r
+ [18.7.68.34])\r
+ by olra.theworths.org (Postfix) with ESMTP id 03E3F431FAE\r
+ for <notmuch@notmuchmail.org>; Fri, 9 Nov 2012 21:29:50 -0800 (PST)\r
+X-AuditID: 12074422-b7f746d0000008cc-62-509de64e33f1\r
+Received: from mailhub-auth-1.mit.edu ( [18.9.21.35])\r
+ by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 9F.33.02252.E46ED905; Sat, 10 Nov 2012 00:29:50 -0500 (EST)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id qAA5TnpI014822; \r
+ Sat, 10 Nov 2012 00:29:50 -0500\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id qAA5Tl59008088\r
+ (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+ Sat, 10 Nov 2012 00:29:49 -0500 (EST)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1TX3db-0000pf-J4; Sat, 10 Nov 2012 00:29:47 -0500\r
+Date: Sat, 10 Nov 2012 00:29:47 -0500\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Mark Walters <markwalters1009@gmail.com>\r
+Subject: Re: [PATCH 0/3] Outline fix for emacs tagging race\r
+Message-ID: <20121110052947.GK22284@mit.edu>\r
+References: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <1352487491-31512-1-git-send-email-markwalters1009@gmail.com>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFupnleLIzCtJLcpLzFFi42IR4hRV1vV7NjfAYMsMcYvVc3ksrt+cyezA\r
+ 5LFz1l12j2erbjEHMEVx2aSk5mSWpRbp2yVwZew5f4W14LB8xa+bKxkbGGdLdjFyckgImEjs\r
+ mbOeEcIWk7hwbz1bFyMXh5DAPkaJzq3XGSGcDYwSR37/ZoZwTjJJdF/fyQrhLGGUmP1oKVg/\r
+ i4CqxOYdb8BsNgENiW37l4PZIgI6ErcPLWAHsZkFpCW+/W5mArGFBawl3t79D1bDC1Qzd9lx\r
+ ZhBbSMBTYk3TCzaIuKDEyZlPWCB6tSRu/HsJ1MsBNmf5Pw6QMKeAl8TL06/AykUFVCSmnNzG\r
+ NoFRaBaS7llIumchdC9gZF7FKJuSW6Wbm5iZU5yarFucnJiXl1qka6qXm1mil5pSuokRFNbs\r
+ Lko7GH8eVDrEKMDBqMTDmxA+N0CINbGsuDL3EKMkB5OSKO/PJ0AhvqT8lMqMxOKM+KLSnNTi\r
+ Q4wSHMxKIryv+oFyvCmJlVWpRfkwKWkOFiVx3mspN/2FBNITS1KzU1MLUotgsjIcHEoSvIVP\r
+ gRoFi1LTUyvSMnNKENJMHJwgw3mAhseA1PAWFyTmFmemQ+RPMepyHH0z9yGjEEtefl6qlDiv\r
+ D0iRAEhRRmke3BxYOnrFKA70ljBvIkgVDzCVwU16BbSECWhJ45E5IEtKEhFSUg2MRZ1vb+yp\r
+ +v2mgV1LtbT/tNq+bbmuOttDFevMa2Z+lFe+Wfl4xopNXkozVQ6EZ78KdpRbc7Ihw7pIZ29p\r
+ 0mrd0yeW/sufa7I3aEfMUZlHp37zHucNsr97e86FUKcZO1xtOP18Shu877Wqu8gzaJveesnl\r
+ uXm26+FQuWs8zDc4VjO+NtMRM9iixFKckWioxVxUnAgAATH5siIDAAA=\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sat, 10 Nov 2012 05:29:53 -0000\r
+\r
+Quoth Mark Walters on Nov 09 at 6:58 pm:\r
+> For a long time [1] there have been two related races in tagging from\r
+> the search buffer.\r
+> \r
+> The first is that when tagging (including archiving) a thread message\r
+> which arrived after the buffer was created may also be tagged. This is\r
+> because the tagging is done based on the thread-id not on the\r
+> individual messages.\r
+> \r
+> The second is when using the '*' command to tag all messages. This is\r
+> not quite the same as this command only tags messages matching the\r
+> query not all messages in all threads that contain a message matching\r
+> the query. Thus if more messages now match than when the buffer was\r
+> created (eg some external tagging script has run) then this command\r
+> can unexpectedly tag these messages too.\r
+> \r
+> One solution that was discussed in [2] was for the search output of\r
+> notmuch to include the message-ids of both matching and non-matching\r
+> messages. At that time that was difficult to implement as it was\r
+> unclear how to escape the message ids when using the text\r
+> format. Since emacs now uses JSON for search mode this problem is\r
+> solved.\r
+> \r
+> This patch series implements the above mentioned solution and seems to\r
+> work except for one problem.\r
+> \r
+> Since emacs now tags each message in a thread in the search buffer it\r
+> is easy to ask it to tag a lot of messages. This could be done\r
+> individually which would be ridiculously slow so instead they are all\r
+> done in one batch. But now it is relatively easy to take notmuch over\r
+> the threshold for ARG_MAX.\r
+> \r
+> In [3] Tomi did some experiments and found on a standard Debian system\r
+> with getconf ARG_MAX =131072 that command lines with 10000 200 byte\r
+> arguments worked. I am a little puzzled by that as I get the same\r
+> results and I getconf ARG_MAX gives 2097152 for me.\r
+> \r
+> More importantly though, when trying to execute a command from emacs I\r
+> am finding that 131072 is the limit on the command length in bytes and\r
+> we can hit this with something around 1500 messages (see end for a\r
+> very hacky emacs test script). This is probably more than we can\r
+> expect in a single thread so tagging from show is probably safe but it\r
+> does cause a problem when tagging from search.\r
+> \r
+> I can think of several possible solutions (e.g., batch it in my new\r
+> stuff, put some batching in notmuch-tag, all notmuch tag to read a\r
+> query from stdin). But before any larger more intrusive change do\r
+> people like the general approach? Does anyone have a good way to get\r
+> round the command line size problem?\r
+> \r
+> Best wishes \r
+> \r
+> Mark\r
+> \r
+> \r
+> [1] id:87ocmtg9ni.fsf@yoom.home.cworth.org\r
+> [2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=UN7McECrg@mail.gmail.com\r
+> [3] id:m2liody7av.fsf@guru.guru-group.fi\r
+\r
+I'm glad to see someone picking up this bug. Besides somehow dealing\r
+with long command-lines, when we were last exploring this race, I had\r
+found that it was 3-4x more efficient to use Xapian document IDs\r
+directly rather than message IDs [1]. It's probably best *not* to do\r
+this initially for the sake of simplicity, but I think a simple tweak\r
+to your approach would let us seamlessly transition to this in the\r
+future. Rather than extending the JSON output with what are\r
+explicitly message IDs, instead extend the output with opaque queries\r
+that are guaranteed to match the matching/non-matching messages in the\r
+thread and are guaranteed to be combinable, but aren't guaranteed to\r
+be of any particular form. For now, the CLI can simply output id:\r
+queries for these, but in the future we could easily add a special\r
+query syntax for docid queries or, if we ever move to a custom query\r
+parser, support docids in any query.\r
+\r
+For the long command line problem, one easy solution is to support,\r
+say, '-' as a query syntax that means "read the query from stdin."\r
+This would be a simple addition to query_string_from_args and would\r
+work across the CLI.\r
+\r
+[1] id:CAH-f9WsPj=1Eu=g3sOePJgCTBFs6HrLdLq18xMEnJ8aZ00yCEg@mail.gmail.com\r
projects / notmuch-archives.git / commitdiff