From 16677ccc037ad4f9dc6dd72c4ce70ae19095aa33 Mon Sep 17 00:00:00 2001 From: Austin Clements Date: Fri, 6 Jul 2012 20:29:41 +2000 Subject: [PATCH] Re: [PATCH v2 0/9] JSON-based search-mode --- fb/37e86373a8fcb31ea0a63392ebd36275e66447 | 299 ++++++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 fb/37e86373a8fcb31ea0a63392ebd36275e66447 diff --git a/fb/37e86373a8fcb31ea0a63392ebd36275e66447 b/fb/37e86373a8fcb31ea0a63392ebd36275e66447 new file mode 100644 index 000000000..ff4883d24 --- /dev/null +++ b/fb/37e86373a8fcb31ea0a63392ebd36275e66447 @@ -0,0 +1,299 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 06E74431FB6 + for ; Thu, 5 Jul 2012 17:29:49 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id qi5oD58q+Za7 for ; + Thu, 5 Jul 2012 17:29:48 -0700 (PDT) +Received: from dmz-mailsec-scanner-3.mit.edu (DMZ-MAILSEC-SCANNER-3.MIT.EDU + [18.9.25.14]) + by olra.theworths.org (Postfix) with ESMTP id E6BE1431FAE + for ; Thu, 5 Jul 2012 17:29:47 -0700 (PDT) +X-AuditID: 1209190e-b7fb56d0000008b2-b4-4ff6317a344e +Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) + by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP + id 9E.60.02226.A7136FF4; Thu, 5 Jul 2012 20:29:46 -0400 (EDT) +Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) + by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id q660Tjd2002667; + Thu, 5 Jul 2012 20:29:45 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q660Tfn6015427 + (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); + Thu, 5 Jul 2012 20:29:42 -0400 (EDT) +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77) + (envelope-from ) + id 1SmwQX-00057a-77; Thu, 05 Jul 2012 20:29:41 -0400 +Date: Thu, 5 Jul 2012 20:29:41 -0400 +From: Austin Clements +To: Mark Walters +Subject: Re: [PATCH v2 0/9] JSON-based search-mode +Message-ID: <20120706002941.GB18195@mit.edu> +References: <1341354059-29396-1-git-send-email-amdragon@mit.edu> + <1341521547-15502-1-git-send-email-amdragon@mit.edu> + <87y5mx3mtd.fsf@qmul.ac.uk> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <87y5mx3mtd.fsf@qmul.ac.uk> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFprJKsWRmVeSWpSXmKPExsUixCmqrVtl+M3f4HSLlMXquTwW12/OZLZ4 + s3IeqwOzx85Zd9k9Dn9dyOLxbNUt5gDmKC6blNSczLLUIn27BK6M/S0XGAv+elQsmXuRuYFx + l0UXIyeHhICJRP+GvYwQtpjEhXvr2boYuTiEBPYxShy+cwvKWc8o8WPHb3YI5wSTRFfzWlYI + ZwmjRPfq6yxdjBwcLAIqEm/P14CMYhPQkNi2fznYWBEBHYnbhxawg9jMAvoSK0/OZAaxhQWM + JPY17GYBsXmBaua1/IeaOZVRYvrOdjaIhKDEyZlPWCCatSRu/HvJBLKLWUBaYvk/DpAwJ9Cu + 3T/uge0SBTphysltbBMYhWYh6Z6FpHsWQvcCRuZVjLIpuVW6uYmZOcWpybrFyYl5ealFusZ6 + uZkleqkppZsYwaEuybeD8etBpUOMAhyMSjy8xrlf/IVYE8uKK3MPMUpyMCmJ8j7W/+YvxJeU + n1KZkVicEV9UmpNafIhRgoNZSYS3N+OrvxBvSmJlVWpRPkxKmoNFSZz3SspNfyGB9MSS1OzU + 1ILUIpisDAeHkgTvSgOgoYJFqempFWmZOSUIaSYOTpDhPEDDF4LU8BYXJOYWZ6ZD5E8x6nKs + e3PkBqMQS15+XqqUOO9qkCIBkKKM0jy4ObAU9YpRHOgtYd61IFU8wPQGN+kV0BImoCV5iz+B + LClJREhJNTCGf+rov/p+xbnY8yvTZqWwHTr3vrN9hjZvlemxwDuqhlLc58/y+ykfjj99e9Ep + hw1Xbt3avOHlhr+/eae1+ujpTlt68Naqz4WvsxW3nBbwDmD1+/vYLqDjwJYvf66xzV7NJXI/ + 85FI+oETjR3fJv8Oy9/9dlvp62cuFjoL9hncV1isF+oR2HfQXYmlOCPRUIu5qDgRACGTIdYs AwAA +Cc: tomi.ollila@iki.fi, notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Fri, 06 Jul 2012 00:29:49 -0000 + +Quoth Mark Walters on Jul 05 at 10:44 pm: +> On Thu, 05 Jul 2012, Austin Clements wrote: +> > This should account for all of Mark's and Tomi's comments. This +> > version +> > * renames the "format" variables to "format-string" and "spec" to be +> > less confusing, +> > * reverts to the original behavior of ignoring the user's format +> > specification for tags (since we make assumptions about this format +> > elsewhere), +> > * swaps the error helper and search-target patches to fix the +> > temporary issue with error message placement, +> > * adds documentation on point movement in the JSON parser, +> > * breaks out the JSON EOF testing function, +> > * beefs up a few commit messages, +> > * and adds a NEWS patch. +> > +> > For ease of reviewing, the diff diff is below. +> > +> > I've written most of a follow-on patch series that cleans up the +> > handling of metadata and tag changes by attaching the JSON result +> > object to the result. This means we don't need a proliferation of +> > text properties to store the result metadata, and we can make updates +> > to a result (e.g., tag changes) by updating this result object and +> > then re-rendering the result line from scratch, rather than trying to +> > update the result line in place. This makes it possible to obey user +> > formatting for the tag list and has other perks like recoloring +> > results when their tags change. I'll send it along once this patch +> > series is accepted. +> > +> > diff --git a/NEWS b/NEWS +> > index d29ec5b..a1a6e93 100644 +> > --- a/NEWS +> > +++ b/NEWS +> > @@ -14,6 +14,23 @@ Maildir tag synchronization +> > messages (typically causing new messages to not receive the "unread" +> > tag). +> > +> > +Emacs Interface +> > +--------------- +> > + +> > +Search now uses the JSON format internally +> > + +> > + This should address problems with unusual characters in authors and +> > + subject lines that could confuse the old text-based search parser. +> > + +> > +The date shown in search results is no longer padded before applying +> > +user-specified formatting +> > + +> > + Previously, the date in the search results was padded to fixed width +> > + before being formatted with `notmuch-search-result-format`. It is +> > + no longer padded. The default format has been updated, but if +> > + you've customized this variable, you may have to change your date +> > + format from `"%s "` to `"%12s "`. +> > + +> > Notmuch 0.13.2 (2012-06-02) +> > =========================== +> > +> > diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el +> > index f7cda33..9e04d97 100644 +> > --- a/emacs/notmuch-lib.el +> > +++ b/emacs/notmuch-lib.el +> > @@ -399,8 +399,9 @@ resynchronize after an error by moving point." +> > +> > Returns 'retry if there is insufficient input to parse the +> > beginning of the compound. If this is able to parse the +> > -beginning of a compound, it returns t and later calls to +> > -`notmuch-json-read' will return the compound's elements. +> > +beginning of a compound, it moves point past the token that opens +> > +the compound and returns t. Later calls to `notmuch-json-read' +> > +will return the compound's elements. +> > +> > Entering JSON objects is current unimplemented." +> > +> > @@ -423,7 +424,8 @@ Entering JSON objects is current unimplemented." +> > Returns 'retry if there is insufficient input to parse a complete +> > JSON value. If the parser is currently inside a compound value +> > and the next token ends the list or object, returns 'end. +> > -Otherwise, returns the value." +> > +Otherwise, moves point to just past the end of the value and +> > +returns the value." +> +> I think that point can move when 'retry is returned (past terminators +> and commas for example). It might also be worth saying that it returns +> the next value passing command and terminators and whitespace after +> point. + +How about + +Returns 'retry if there is insufficient input to parse a complete JSON +value (though it may still move point over separators or whitespace). +If the parser is currently inside a compound value and the next token +ends the list or object, this moves point just past the terminator and +returns 'end. Otherwise, this moves point to just past the end of the +value and returns the value. + +? + +> > +> > (with-current-buffer (notmuch-json-buffer jp) +> > (or +> > @@ -474,11 +476,22 @@ Otherwise, returns the value." +> > (notmuch-json-partial-pos jp) nil +> > (notmuch-json-partial-state jp) nil) +> > ;; Parse the value +> > - (let* ((json-object-type 'plist) +> > - (json-array-type 'list) +> > - (json-false nil)) +> > + (let ((json-object-type 'plist) +> > + (json-array-type 'list) +> > + (json-false nil)) +> > (json-read))))))) +> > +> > +(defun notmuch-json-eof (jp) +> > + "Signal a json-error if there is more input in JP's buffer. +> +> Would `data' be better than `input' (to distinguish allowed whitespace +> from disallowed data)? + +Ah, yes. + + "Signal a json-error if there is more data in JP's buffer. + +Moves point to the beginning of any trailing data or to the end +of the buffer if there is only trailing whitespace." + +? + +> > +Moves point to the beginning of any trailing garbage or to the +> > +end of the buffer if there is no trailing garbage." +> > + +> > + (with-current-buffer (notmuch-json-buffer jp) +> > + (skip-chars-forward " \t\r\n") +> > + (unless (eobp) +> > + (signal 'json-error (list "Trailing garbage following JSON data"))))) +> > + +> > (provide 'notmuch-lib) +> > +> > ;; Local Variables: +> > diff --git a/emacs/notmuch.el b/emacs/notmuch.el +> > index 2a09a98..fabb7c0 100644 +> > --- a/emacs/notmuch.el +> > +++ b/emacs/notmuch.el +> > @@ -702,28 +702,29 @@ non-authors is found, assume that all of the authors match." +> > (overlay-put overlay 'isearch-open-invisible #'delete-overlay))) +> > (insert padding)))) +> > +> > -(defun notmuch-search-insert-field (field format result) +> > +(defun notmuch-search-insert-field (field format-string result) +> > (cond +> > ((string-equal field "date") +> > - (insert (propertize (format format (plist-get result :date_relative)) +> > + (insert (propertize (format format-string (plist-get result :date_relative)) +> > 'face 'notmuch-search-date))) +> > ((string-equal field "count") +> > - (insert (propertize (format format (format "[%s/%s]" +> > - (plist-get result :matched) +> > - (plist-get result :total))) +> > + (insert (propertize (format format-string +> > + (format "[%s/%s]" (plist-get result :matched) +> > + (plist-get result :total))) +> > 'face 'notmuch-search-count))) +> > ((string-equal field "subject") +> > - (insert (propertize (format format (plist-get result :subject)) +> > + (insert (propertize (format format-string (plist-get result :subject)) +> > 'face 'notmuch-search-subject))) +> > +> > ((string-equal field "authors") +> > - (notmuch-search-insert-authors format (plist-get result :authors))) +> > + (notmuch-search-insert-authors format-string (plist-get result :authors))) +> > +> > ((string-equal field "tags") +> > - (insert +> > - (format format (propertize +> > - (mapconcat 'identity (plist-get result :tags) " ") +> > - 'font-lock-face 'notmuch-tag-face)))))) +> > + ;; Ignore format-string here because notmuch-search-set-tags +> > + ;; depends on the format of this +> > + (insert (concat "(" (propertize +> > + (mapconcat 'identity (plist-get result :tags) " ") +> > + 'font-lock-face 'notmuch-tag-face) ")"))))) +> > +> > (defun notmuch-search-show-result (result) +> > ;; Ignore excluded matches +> > @@ -731,8 +732,8 @@ non-authors is found, assume that all of the authors match." +> > (let ((beg (point-max))) +> > (save-excursion +> > (goto-char beg) +> > - (dolist (format notmuch-search-result-format) +> > - (notmuch-search-insert-field (car format) (cdr format) result)) +> > + (dolist (spec notmuch-search-result-format) +> > + (notmuch-search-insert-field (car spec) (cdr spec) result)) +> > (insert "\n") +> > (notmuch-search-color-line beg (point) (plist-get result :tags)) +> > (put-text-property beg (point) 'notmuch-search-thread-id +> > @@ -790,11 +791,8 @@ non-authors is found, assume that all of the authors match." +> > (otherwise (notmuch-search-show-result result))))) +> > ((end) +> > ;; Any trailing data is unexpected +> > - (with-current-buffer parse-buf +> > - (skip-chars-forward " \t\r\n") +> > - (if (eobp) +> > - (setq done t) +> > - (signal 'json-error nil))))) +> > + (notmuch-json-eof notmuch-search-json-parser) +> > + (setq done t))) +> +> I think this used to only set `done' if there was not trailing data but +> now does so anyway? + +It still won't set done if there's trailing data because +notmuch-json-eof will signal an error, which will unwind to the +condition-case (which will then consume said trailing data and done +will get set on the next time through the loop). + +> Best wishes +> +> Mark +> +> > (json-error +> > ;; Do our best to resynchronize and ensure forward +> > ;; progress -- 2.26.2