--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 6CE17431FB6\r
+ for <notmuch@notmuchmail.org>; Tue, 25 Dec 2012 19:49:03 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id Z-+Z449FvnOD for <notmuch@notmuchmail.org>;\r
+ Tue, 25 Dec 2012 19:48:58 -0800 (PST)\r
+Received: from dmz-mailsec-scanner-5.mit.edu (DMZ-MAILSEC-SCANNER-5.MIT.EDU\r
+ [18.7.68.34])\r
+ by olra.theworths.org (Postfix) with ESMTP id 95D45431FAF\r
+ for <notmuch@notmuchmail.org>; Tue, 25 Dec 2012 19:48:58 -0800 (PST)\r
+X-AuditID: 12074422-b7f616d000000e7c-18-50da73a9b9d2\r
+Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])\r
+ by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id D1.4B.03708.9A37AD05; Tue, 25 Dec 2012 22:48:57 -0500 (EST)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id qBQ3muYt019978; \r
+ Tue, 25 Dec 2012 22:48:56 -0500\r
+Received: from drake.dyndns.org (c-76-21-105-205.hsd1.ca.comcast.net\r
+ [76.21.105.205]) (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id qBQ3mrZR013180\r
+ (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+ Tue, 25 Dec 2012 22:48:55 -0500 (EST)\r
+Received: from amthrax by drake.dyndns.org with local (Exim 4.77)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1Tnhz8-0002yu-HR; Tue, 25 Dec 2012 22:48:50 -0500\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: notmuch@notmuchmail.org\r
+Subject: [PATCH v2 0/5] Use Xapian query syntax for batch-tag dump/restore\r
+Date: Tue, 25 Dec 2012 22:48:38 -0500\r
+Message-Id: <1356493723-11085-1-git-send-email-amdragon@mit.edu>\r
+X-Mailer: git-send-email 1.7.10.4\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFtrGIsWRmVeSWpSXmKPExsUixG6nrruy+FaAweTDZhY3WrsZLZqmO1us\r
+ nstjcf3mTGYHFo+ds+6ye9y6/5rd49mqW8weWw69Zw5gieKySUnNySxLLdK3S+DKuPL5OUvB\r
+ Q92KGa3TWBoY9yh3MXJySAiYSLx8+ZwZwhaTuHBvPRuILSSwj1HiwU6NLkYuIHsDo8TUr3fZ\r
+ IJyLTBLzpm1nhXDmMkqsP3aQEaSFTUBDYtv+5WC2iIC0xM67s1lBbGYBR4kzr9vA4sICXhJn\r
+ ViwAmsTBwSKgKvHqvyBImFfAQWLmgUtQVyhKdD+bwDaBkXcBI8MqRtmU3Crd3MTMnOLUZN3i\r
+ 5MS8vNQiXVO93MwSvdSU0k2M4OBxUdrB+POg0iFGAQ5GJR7eDd9vBgixJpYVV+YeYpTkYFIS\r
+ 5d1ecCtAiC8pP6UyI7E4I76oNCe1+BCjBAezkgiv80egct6UxMqq1KJ8mJQ0B4uSOO+1lJv+\r
+ QgLpiSWp2ampBalFMFkZDg4lCd7NRUBDBYtS01Mr0jJzShDSTBycIMN5gIbfBKnhLS5IzC3O\r
+ TIfIn2JUlBLnvQCSEABJZJTmwfXCovsVozjQK8K8e0CqeICJAa77FdBgJqDBsXw3QAaXJCKk\r
+ pBoYZ89tuXEkKeE0778ivlLzC/9cRS/KR988Of/Srtf7AwXf107VerxWREAlhGHFAa6TN+5s\r
+ FDuw/e/BLVNNql5ItTp+mXelyjl21xOx3MK+/I279csT3ireWcUg8X5LMdstqwdK9i28fCWb\r
+ Tl5cbX/SquNr5+aOxka/3p82bDZe/myWf+4as15PV2Ipzkg01GIuKk4EAGT71V3JAgAA\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 26 Dec 2012 03:49:04 -0000\r
+\r
+This obsoletes\r
+\r
+ id:1356415076-5692-1-git-send-email-amdragon@mit.edu\r
+\r
+In addition to incorporating all of David's suggestions, this reworks\r
+the boolean term parsing so it only handles the subset of quoting\r
+syntax used by make_boolean_term (which also happens to be all that we\r
+described in the man page for the format). The diff from v1 is below.\r
+\r
+diff --git a/man/man1/notmuch-restore.1 b/man/man1/notmuch-restore.1\r
+index 6bba628..78fef52 100644\r
+--- a/man/man1/notmuch-restore.1\r
++++ b/man/man1/notmuch-restore.1\r
+@@ -57,10 +57,8 @@ sup calls them).\r
+ The\r
+ .B batch-tag\r
+ dump format is intended to more robust against malformed message-ids\r
+-and tags containing whitespace or non-\fBascii\fR(7) characters. This\r
+-format hex-escapes all characters those outside of a small character\r
+-set, intended to be suitable for e.g. pathnames in most UNIX-like\r
+-systems.\r
++and tags containing whitespace or non-\fBascii\fR(7) characters. See\r
++\fBnotmuch-dump\fR(1) for details on this format.\r
+ \r
+ .B "notmuch restore"\r
+ updates the maildir flags according to tag changes if the\r
+diff --git a/test/dump-restore b/test/dump-restore\r
+index aecc393..f9ae5b3 100755\r
+--- a/test/dump-restore\r
++++ b/test/dump-restore\r
+@@ -200,6 +200,8 @@ a\r
+ # the next non-comment line should report an an empty tag error for\r
+ # batch tagging, but not for restore\r
+ + +e -- id:20091117232137.GA7669@griffis1.net\r
++# valid id, but warning about missing message\r
+++e id:missing_message_id\r
+ EOF\r
+ \r
+ cat <<EOF > EXPECTED\r
+@@ -211,6 +213,7 @@ Warning: no query string after -- [+c +d --]\r
+ Warning: hex decoding of tag %zz failed [+%zz -- id:whatever]\r
+ Warning: cannot parse query: id:"\r
+ Warning: not an id query: tag:abc\r
++Warning: cannot apply tags to missing message: missing_message_id\r
+ EOF\r
+ \r
+ test_expect_equal_file EXPECTED OUTPUT\r
+diff --git a/test/random-corpus.c b/test/random-corpus.c\r
+index d0e3e8f..8b7748e 100644\r
+--- a/test/random-corpus.c\r
++++ b/test/random-corpus.c\r
+@@ -96,9 +96,9 @@ random_utf8_string (void *ctx, size_t char_count)\r
+ buf = talloc_realloc (ctx, buf, gchar, buf_size);\r
+ }\r
+ \r
+- randomchar = random_unichar ();\r
+- if (randomchar == '\n')\r
+- randomchar = 'x';\r
++ do {\r
++ randomchar = random_unichar ();\r
++ } while (randomchar == '\n');\r
+ \r
+ written = g_unichar_to_utf8 (randomchar, buf + offset);\r
+ \r
+diff --git a/util/string-util.c b/util/string-util.c\r
+index eaa6c99..db01b4b 100644\r
+--- a/util/string-util.c\r
++++ b/util/string-util.c\r
+@@ -43,9 +43,11 @@ make_boolean_term (void *ctx, const char *prefix, const char *term,\r
+ size_t needed = 3;\r
+ int need_quoting = 0;\r
+ \r
+- /* Do we need quoting? */\r
++ /* Do we need quoting? To be paranoid, we quote anything\r
++ * containing a quote, even though it only matters at the\r
++ * beginning, and anything containing non-ASCII text. */\r
+ for (in = term; *in && !need_quoting; in++)\r
+- if (*in <= ' ' || *in == ')' || *in == '"')\r
++ if (*in <= ' ' || *in == ')' || *in == '"' || (unsigned char)*in > 127)\r
+ need_quoting = 1;\r
+ \r
+ if (need_quoting)\r
+@@ -95,21 +97,6 @@ make_boolean_term (void *ctx, const char *prefix, const char *term,\r
+ return 0;\r
+ }\r
+ \r
+-static int\r
+-consume_double_quote (const char **str)\r
+-{\r
+- if (**str == '"') {\r
+- ++*str;\r
+- return 1;\r
+- } else if (strncmp(*str, "\xe2\x80\x9c", 3) == 0 || /* UTF8 0x201c */\r
+- strncmp(*str, "\xe2\x80\x9d", 3) == 0) { /* UTF8 0x201d */\r
+- *str += 3;\r
+- return 3;\r
+- } else {\r
+- return 0;\r
+- }\r
+-}\r
+-\r
+ int\r
+ parse_boolean_term (void *ctx, const char *str,\r
+ char **prefix_out, char **term_out)\r
+@@ -123,28 +110,31 @@ parse_boolean_term (void *ctx, const char *str,\r
+ *prefix_out = talloc_strndup (ctx, str, pos - str);\r
+ ++pos;\r
+ \r
+- /* Implement Xapian's boolean term de-quoting. This is a nearly\r
+- * direct translation of QueryParser::Internal::parse_query. */\r
+- pos = *term_out = talloc_strdup (ctx, pos);\r
+- if (consume_double_quote (&pos)) {\r
+- char *out = talloc_strdup (ctx, pos);\r
+- pos = *term_out = out;\r
+- while (1) {\r
+- if (! *pos) {\r
+- /* Premature end of string */\r
+- goto FAIL;\r
+- } else if (*pos == '"') {\r
+- if (*++pos != '"')\r
++ /* Implement de-quoting compatible with make_boolean_term. */\r
++ if (*pos == '"') {\r
++ char *out = talloc_strdup (ctx, pos + 1);\r
++ int closed = 0;\r
++ /* Find the closing quote and un-double doubled internal\r
++ * quotes. */\r
++ for (pos = *term_out = out; *pos; ) {\r
++ if (*pos == '"') {\r
++ ++pos;\r
++ if (*pos != '"') {\r
++ /* Found the closing quote. */\r
++ closed = 1;\r
+ break;\r
+- } else if (consume_double_quote (&pos)) {\r
+- break;\r
++ }\r
+ }\r
+ *out++ = *pos++;\r
+ }\r
+- if (*pos)\r
++ /* Did the term terminate without a closing quote or is there\r
++ * trailing text after the closing quote? */\r
++ if (!closed || *pos)\r
+ goto FAIL;\r
+ *out = '\0';\r
+ } else {\r
++ *term_out = talloc_strdup (ctx, pos);\r
++ /* Check for text after the boolean term. */\r
+ while (*pos > ' ' && *pos != ')')\r
+ ++pos;\r
+ if (*pos)\r
+diff --git a/util/string-util.h b/util/string-util.h\r
+index e4e4c42..aff2d65 100644\r
+--- a/util/string-util.h\r
++++ b/util/string-util.h\r
+@@ -28,9 +28,9 @@ char *strtok_len (char *s, const char *delim, size_t *len);\r
+ int make_boolean_term (void *talloc_ctx, const char *prefix, const char *term,\r
+ char **buf, size_t *len);\r
+ \r
+-/* Parse a boolean term query, returning the prefix in *prefix_out and\r
+- * the term in *term_out. *prefix_out and *term_out will be talloc'd\r
+- * with context ctx.\r
++/* Parse a boolean term query produced by make_boolean_term, returning\r
++ * the prefix in *prefix_out and the term in *term_out. *prefix_out\r
++ * and *term_out will be talloc'd with context ctx.\r
+ *\r
+ * Return: 0 on success, non-zero on parse error (including trailing\r
+ * data in str).\r
+\r
+\r
projects / notmuch-archives.git / commitdiff