Re: `notmuch-escape-boolean-term': Broken for non-ascii characters

diff --git a/e8/a21f1fbd5f78c4e4f22796671b2b6927e28931 b/e8/a21f1fbd5f78c4e4f22796671b2b6927e28931
new file mode 100644 (file)
index 0000000..89fd925
--- /dev/null
+++ b/e8/a21f1fbd5f78c4e4f22796671b2b6927e28931
@@ -0,0 +1,106 @@
+Return-Path: <aclements@csail.mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 17787431FBC\r
+       for <notmuch@notmuchmail.org>; Tue, 12 Aug 2014 07:33:11 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id dJLrDEuSwQPP for <notmuch@notmuchmail.org>;\r
+       Tue, 12 Aug 2014 07:33:03 -0700 (PDT)\r
+Received: from outgoing.csail.mit.edu (outgoing.csail.mit.edu [128.30.2.149])\r
+       by olra.theworths.org (Postfix) with ESMTP id B7346431FAF\r
+       for <notmuch@notmuchmail.org>; Tue, 12 Aug 2014 07:33:03 -0700 (PDT)\r
+Received: from webmail.csail.mit.edu ([128.30.2.164] helo=webmail)\r
+       by outgoing.csail.mit.edu with esmtpsa\r
+       (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72)\r
+       (envelope-from <aclements@csail.mit.edu>)\r
+       id 1XHD8G-0007qi-Rh; Tue, 12 Aug 2014 10:33:00 -0400\r
+Received: from 216-15-114-40.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com\r
+       (216-15-114-40.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com [216.15.114.40])\r
+       by webmail.csail.mit.edu (Horde Framework) with HTTP;\r
+       Tue, 12 Aug 2014 10:33:00 -0400\r
+Date: Tue, 12 Aug 2014 10:33:00 -0400\r
+Message-ID:\r
+ <20140812103300.Horde.O1lIjfCL-Lh8XGn65RO2Cg1@webmail.csail.mit.edu>\r
+From: "Austin T. Clements" <aclements@csail.mit.edu>\r
+To: Moritz Ulrich <moritz@tarn-vedra.de>\r
+Subject: Re: `notmuch-escape-boolean-term': Broken for non-ascii characters\r
+References: <874mxiu5hj.fsf@tarn-vedra.de>\r
+In-Reply-To: <874mxiu5hj.fsf@tarn-vedra.de>\r
+User-Agent: Internet Messaging Program (IMP) H5 (6.1.4)\r
+Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes\r
+MIME-Version: 1.0\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: 8bit\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 12 Aug 2014 14:33:11 -0000\r
+\r
+Quoting Moritz Ulrich <moritz@tarn-vedra.de>:\r
+> Hello,\r
+>\r
+> I recently adopted notmuch as my primary way to read mail, so thank you\r
+> for this great tool!\r
+>\r
+> Unfortunately, I ran into a problem of the Emacs side of the project\r
+> when used in a non-ascii environment:\r
+>\r
+> Having a tag named 'uni-köln', the tag:-completion doesn't work.\r
+>\r
+> This is caused by `notmuch-escape-boolean-term' errornously escaping the\r
+> above string:\r
+>\r
+> (notmuch-escape-boolean-term "uni-köln") => "\"uni-köln\""\r
+>\r
+> This is caused by `string-match' with the following errornously matching\r
+> my tag:\r
+>\r
+> (string-match "[^!#-'*-~]" "uni-köln") => 5\r
+> (string-match "[^!#-'*-~]" "uni-koln") => nil\r
+>\r
+> I'm not exactly sure how to tackle this - the Regexp was crafted to match\r
+> (, ), " if I understand it correct. A simple way would be just adding\r
+> more characters as a sort-of whitelist. A nicer solution would be\r
+> converting it from [^...] to [...] to explicitly mark letters that needs\r
+> to be escaped.\r
+\r
+notmuch-escape-boolean-term used to use a blacklist, but we switched\r
+to a whitelist because Xapian's own parser has changed over the years\r
+in its handling of non-ASCII characters and invalidated our blacklist.\r
+Ultimately it seemed much safer to go with a whitelist.  Quoting\r
+"uni-köln" isn't erroneous, it's just conservative.\r
+\r
+Could you explain in more detail what's broken?  I tried adding the\r
+tag uni-köln to a message in Emacs, then hitting "s" to start a search\r
+then "tag:<TAB>" and that tag (surrounded by quotes) was one of the\r
+completion options.  Upon completing to that tag, the search worked\r
+fine.\r
+\r
+Are you objecting to the unnecessary (but legal) quotes in the\r
+completion?  We might be able to include Unicode word characters in\r
+the quoting whitelist, though that seems like a spot fix (probably a\r
+fairly broad one, so maybe that's fine) and might be tricky because of\r
+Emacs' somewhat weird Unicode regexp support (using [[:alpha:]] might\r
+Just Work, but we'd have to be careful of the active syntax table).\r
+Or tab completion could recognize that, say, tag:uni doesn't require\r
+quoting, but still expand it to tag:"uni-köln".\r
+\r
+\r