1 Return-Path: <amdragon@mit.edu>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by olra.theworths.org (Postfix) with ESMTP id EE5BF431FD0
\r
6 for <notmuch@notmuchmail.org>; Wed, 20 Jul 2011 13:50:24 -0700 (PDT)
\r
7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
\r
11 X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5
\r
12 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled
\r
13 Received: from olra.theworths.org ([127.0.0.1])
\r
14 by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
\r
15 with ESMTP id 27QeVyQwmBVx for <notmuch@notmuchmail.org>;
\r
16 Wed, 20 Jul 2011 13:50:22 -0700 (PDT)
\r
17 Received: from dmz-mailsec-scanner-5.mit.edu (DMZ-MAILSEC-SCANNER-5.MIT.EDU
\r
19 by olra.theworths.org (Postfix) with ESMTP id 8D394431FB6
\r
20 for <notmuch@notmuchmail.org>; Wed, 20 Jul 2011 13:50:22 -0700 (PDT)
\r
21 X-AuditID: 12074422-b7ba7ae000000a14-a6-4e273f7e061b
\r
22 Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])
\r
23 by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP
\r
24 id 1D.A7.02580.E7F372E4; Wed, 20 Jul 2011 16:50:06 -0400 (EDT)
\r
25 Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])
\r
26 by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id p6KKoLmr011903;
\r
27 Wed, 20 Jul 2011 16:50:21 -0400
\r
28 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])
\r
29 (authenticated bits=0)
\r
30 (User authenticated as amdragon@ATHENA.MIT.EDU)
\r
31 by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id p6KKoJHH001367
\r
32 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);
\r
33 Wed, 20 Jul 2011 16:50:20 -0400 (EDT)
\r
34 Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.72)
\r
35 (envelope-from <amdragon@mit.edu>)
\r
36 id 1QjdiZ-0007Ee-Tb; Wed, 20 Jul 2011 16:50:07 -0400
\r
37 Date: Wed, 20 Jul 2011 16:50:07 -0400
\r
38 From: Austin Clements <amdragon@MIT.EDU>
\r
39 To: Pieter Praet <pieter@praet.org>
\r
40 Subject: JSON parsing performance (was Re: [PATCH v2] emacs: bad regexp @
\r
41 `notmuch-search-process-filter')
\r
42 Message-ID: <20110720205007.GB21316@mit.edu>
\r
43 References: <20110705214234.GA15360@mit.edu>
\r
44 <1310416993-31031-1-git-send-email-pieter@praet.org>
\r
45 <20110711210532.GC25558@mit.edu> <878vs28dvo.fsf@praet.org>
\r
46 <20110713185721.GI25558@mit.edu>
\r
48 Content-Type: multipart/mixed; boundary="/04w6evG8XlLl3ft"
\r
49 Content-Disposition: inline
\r
50 In-Reply-To: <20110713185721.GI25558@mit.edu>
\r
51 User-Agent: Mutt/1.5.20 (2009-06-14)
\r
52 X-Brightmail-Tracker:
\r
53 H4sIAAAAAAAAA+NgFvrIKsWRmVeSWpSXmKPExsUixG6nrltnr+5ncHmrqsW+O1uYLK7fnMls
\r
54 8fv1DWaLpb92szmweOx6/pfJY+esu+wez1bdYvbo2HeZNYAlissmJTUnsyy1SN8ugStj5u6n
\r
55 7AWr/zBW3G84zdbAuG4PYxcjJ4eEgInExHvv2SFsMYkL99azdTFycQgJ7GOUWLjyMyuEs4FR
\r
56 4uOOT8wQzkkmiaU3DkFlljBKfF67C6yfRUBV4vfJNiYQm01AQ2Lb/uVgO0QElCVOP/kJVMPB
\r
57 wSxQKLHkWiVIWFggR6Lh4VdmEJtXQEdiz6t1YLaQwDFGiX1LYiDighInZz5hAbGZBawkfv6Z
\r
58 wAgxRlpi+T8OEJNTQFdi6aR4kApRARWJa/vb2SYwCs1C0jwLSfMshGaIsJbEjX8vmXAKg9gW
\r
59 Ej9+P2XDFDeW2HZ4LuMCRs5VjLIpuVW6uYmZOcWpybrFyYl5ealFuqZ6uZkleqkppZsYwfHo
\r
60 orSD8edBpUOMAhyMSjy8DnzqfkKsiWXFlbmHGCU5mJREeafaAYX4kvJTKjMSizPii0pzUosP
\r
61 MUpwMCuJ8Cr+U/MT4k1JrKxKLcqHSUlzsCiJ85Z4//cVEkhPLEnNTk0tSC2CycpwcChJ8IoD
\r
62 046QYFFqempFWmZOCUKaiYMTZDgP0HBtkBre4oLE3OLMdIj8KUZjjvu37x9h5Lj+FEgKseTl
\r
63 56VKifO+ALlRAKQ0ozQPbhospb5iFAd6Tpj3DUgVDzAdw817BbSKCWhVi7oqyKqSRISUVANj
\r
64 4l4bltdqfb++PTWNKGasSf1lweD64LSoZtcaGZ+0moNHrX7bfWbmm+n0N3B5c2ClCLPYH/W3
\r
65 J5SClDctN1juzrCkj7Exk9lnf4dKlskdZ/nHbdH/Hkx+4rclyG/RLSVb3y1pO6ojMxWsrV36
\r
66 5mrU8pW/0O7YdK6Gb+fToylbzZ4bxbILzlBiKc5INNRiLipOBACOvzSrhAMAAA==
\r
67 Cc: servilio <servilio@gmail.com>, Notmuch Mail <notmuch@notmuchmail.org>,
\r
68 David Edmondson <dme@dme.org>
\r
69 X-BeenThere: notmuch@notmuchmail.org
\r
70 X-Mailman-Version: 2.1.13
\r
72 List-Id: "Use and development of the notmuch mail system."
\r
73 <notmuch.notmuchmail.org>
\r
74 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
\r
75 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
76 List-Archive: <http://notmuchmail.org/pipermail/notmuch>
\r
77 List-Post: <mailto:notmuch@notmuchmail.org>
\r
78 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
79 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
\r
80 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
81 X-List-Received-Date: Wed, 20 Jul 2011 20:50:25 -0000
\r
85 Content-Type: text/plain; charset=us-ascii
\r
86 Content-Disposition: inline
\r
88 Quoth myself on Jul 13 at 2:57 pm:
\r
89 > Quoth Pieter Praet on Jul 13 at 4:16 pm:
\r
90 > > Jamie Zawinski once said/wrote [1]:
\r
91 > > 'Some people, when confronted with a problem, think "I know,
\r
92 > > I'll use regular expressions." Now they have two problems.'
\r
94 > > With this in mind, I set out to get rid of this whole regex mess altogether,
\r
95 > > by populating the search buffer using Notmuch's JSON output instead of doing
\r
96 > > brittle text matching tricks.
\r
98 > > Looking for some documentation, I stumbled upon a long-forgotten gem [2].
\r
100 > > David's already done pretty much all of the work for us!
\r
102 > Yes, similar thoughts were running through my head as I futzed with
\r
103 > the formatting for this. My concern with moving to JSON for search
\r
104 > buffers is that parsing it is about *30 times slower* than the current
\r
105 > regexp-based approach (0.6 seconds versus 0.02 seconds for a mere 1413
\r
106 > result search buffer). I think JSON makes a lot of sense for show
\r
107 > buffers because there's generally less data and it has a lot of
\r
108 > complicated structure. Search results, on the other hand, have a very
\r
109 > simple, regular, and constrained structure, so JSON doesn't buy us
\r
112 > JSON is hard to parse because, like the text search output, it's
\r
113 > designed for human consumption (of course, unlike the text search
\r
114 > output, it's also designed for computer consumption). There's
\r
115 > something to be said for the debuggability and generality of this and
\r
116 > JSON is very good for exchanging small objects, but it's a remarkably
\r
117 > inefficient way to exchange large amounts of data between two
\r
120 > I guess what I'm getting at, though it pains me to say it, is perhaps
\r
121 > search needs a fast, computer-readable interchange format. The
\r
122 > structure of the data is so simple and constrained that this could be
\r
123 > altogether trivial.
\r
125 > Or maybe I need a faster computer.
\r
127 Or maybe I need to un-lame my benchmark.
\r
129 TL;DR: We should use JSON for search results, but possibly not the
\r
130 json.el shipped with Emacs.
\r
132 I realized that my text benchmark didn't capture the cost of
\r
133 extracting the match strings. re-search-forward records matches as
\r
134 buffer positions, which don't get realized into strings until you call
\r
135 match-string. Hence, match-string is quite expensive.
\r
137 Also, Emacs' json.el is slow, so I perked it up. My modified json.el
\r
138 is ~3X faster, particularly for string-heavy output like notmuch's.
\r
139 Though now I'm well into the realm of "eq is faster than =" and "M-x
\r
140 disassemble", so unless I missed something big, this is as fast as it
\r
143 While I was still thinking about new IPC formats, I realized that the
\r
144 text format and the Emacs UI are already tightly coupled, so why not
\r
145 go all the way and use S-expressions for IPC? I now think JSON is
\r
146 fast enough to use, but S-expressions still have a certain appeal.
\r
147 They share most of the benefits of JSON; structure and extensibility
\r
148 in particular. Further, while the content of some ad-hoc format could
\r
149 easily diverge from both the text and JSON formats, S-expressions
\r
150 could exactly parallel the JSON content (with a little more
\r
151 abstraction, they could even share the same format code). For kicks,
\r
152 I included an S-expression benchmark. It beats out the text parser by
\r
153 a factor of two and the optimized JSON parser by a factor of three.
\r
155 Here are the results for my 1,413 result search buffer and timeworn
\r
159 --format=text 0.148s 1.00x
\r
160 --format=json 0.598s 4.04x
\r
161 custom json.el 0.209s 1.41x
\r
162 + string keys 0.195s 1.32x
\r
163 S-expressions 0.066s 0.45x
\r
165 I don't have time right now, but next week I might be able to look
\r
166 through and update dme's JSON-based search code.
\r
169 The benchmark and modified json.el are attached.
\r
171 The benchmark is written so you can open it and eval-buffer, then C-x
\r
172 C-e the various calls in the comments. You can either
\r
173 make-text/make-json, or run notmuch manually, pipe the results into
\r
174 files "text" and "json", and open them in Emacs.
\r
176 Please excuse the modified json.el code; it's gone through zero
\r
180 Content-Type: text/plain; charset=us-ascii
\r
181 Content-Disposition: attachment; filename="timeparse.el"
\r
183 (defmacro time-it (repeat &rest body)
\r
184 (declare (indent 1))
\r
185 (when (not (numberp repeat))
\r
188 (let ((start-time (gensym)) (i (gensym)))
\r
189 `(let ((,start-time (get-internal-run-time)))
\r
190 (dotimes (,i ,repeat)
\r
192 (/ (float-time (time-subtract (get-internal-run-time) ,start-time))
\r
197 (defun make-text ()
\r
198 (with-current-buffer (get-buffer-create "text")
\r
200 (call-process "notmuch" nil t nil "search" "--format=text" "--" "tag:x/notmuch")))
\r
202 (defun time-text ()
\r
203 (with-current-buffer "text"
\r
205 (goto-char (point-min))
\r
206 (while (re-search-forward "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" nil t)
\r
207 (let* ((thread-id (match-string 1))
\r
208 (date (match-string 2))
\r
209 (count (match-string 3))
\r
210 (authors (match-string 4))
\r
211 (subject (match-string 5))
\r
212 (tags (match-string 6))
\r
213 (tag-list (if tags (save-match-data (split-string tags)))))
\r
216 (byte-compile 'time-text)
\r
222 (defun load-custom-json ()
\r
223 (byte-compile-file "json.el")
\r
224 (load-file "./json.elc"))
\r
226 (defun make-json ()
\r
227 (with-current-buffer (get-buffer-create "json")
\r
229 (call-process "notmuch" nil t nil "search" "--format=json" "--" "tag:x/notmuch")))
\r
231 (defun time-json (&optional buf)
\r
232 (with-current-buffer (or buf "json")
\r
233 (let ((json-array-type 'list)
\r
234 (json-object-type 'alist)
\r
235 (json-key-type 'symbol))
\r
237 (goto-char (point-min))
\r
238 (dolist (ent (json-read))
\r
239 ;; (Surprisingly, traversing the structure has no noticeable
\r
240 ;; impact to performance)
\r
241 (let ((thread-id (assq 'thread ent))
\r
242 (date (assq 'timestamp ent))
\r
243 (matched (assq 'matched ent))
\r
244 (total (assq 'total ent))
\r
245 (authors (assq 'authors ent))
\r
246 (subject (assq 'subject ent))
\r
247 (tag-list (assq 'tags ent)))
\r
250 (defun time-json-string-keys (&optional buf)
\r
251 (with-current-buffer (or buf "json")
\r
252 (let ((json-array-type 'list)
\r
253 (json-object-type 'alist)
\r
254 (json-key-type 'string))
\r
256 (goto-char (point-min))
\r
257 (dolist (ent (json-read))
\r
258 (let ((thread-id (assoc "thread" ent))
\r
259 (date (assoc "timestamp" ent))
\r
260 (matched (assoc "matched" ent))
\r
261 (total (assoc "total" ent))
\r
262 (authors (assoc "authors" ent))
\r
263 (subject (assoc "subject" ent))
\r
264 (tag-list (assoc "tags" ent)))
\r
267 (byte-compile 'time-json)
\r
268 (byte-compile 'time-json-string-keys)
\r
271 ;; (time-json-string-keys)
\r
272 ;; (load-custom-json)
\r
276 (defun make-sexp ()
\r
277 (with-current-buffer (get-buffer-create "sexp")
\r
280 (with-current-buffer "json"
\r
281 (let ((json-array-type 'list)
\r
282 (json-object-type 'alist)
\r
283 (json-key-type 'symbol))
\r
284 (goto-char (point-min))
\r
286 (get-buffer "sexp"))
\r
289 (defun time-sexp ()
\r
290 (with-current-buffer "sexp"
\r
291 (let ((buf (current-buffer)))
\r
292 (time-it 10 (goto-char (point-min)) (read buf)))))
\r
294 (byte-compile 'time-sexp)
\r
300 (defun make-packed-json ()
\r
301 (let ((buf (get-buffer-create "packed-json")))
\r
302 (with-current-buffer "json"
\r
303 (copy-to-buffer buf (point-min) (point-max)))
\r
304 (with-current-buffer buf
\r
305 (while (re-search-forward "^\\([^\"]*\"[^\"]+\"\\): \\([[\"0-9]\\)" nil t)
\r
306 (replace-match "\\1:\\2" nil nil))
\r
307 (goto-char (point-min))
\r
308 (while (re-search-forward "\\([\"0-9]\\),\n" nil t)
\r
309 (replace-match "\\1," nil nil)))))
\r
311 (defun time-packed-json ()
\r
312 (time-json "packed-json"))
\r
314 ;; (make-packed-json)
\r
315 ;; (time-packed-json)
\r
318 Content-Type: text/plain; charset=us-ascii
\r
319 Content-Disposition: attachment; filename="json.el"
\r
321 ;;; json.el --- JavaScript Object Notation parser / generator
\r
323 ;; Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
\r
325 ;; Author: Edward O'Connor <ted@oconnor.cx>
\r
327 ;; Keywords: convenience
\r
329 ;; This file is part of GNU Emacs.
\r
331 ;; GNU Emacs is free software: you can redistribute it and/or modify
\r
332 ;; it under the terms of the GNU General Public License as published by
\r
333 ;; the Free Software Foundation, either version 3 of the License, or
\r
334 ;; (at your option) any later version.
\r
336 ;; GNU Emacs is distributed in the hope that it will be useful,
\r
337 ;; but WITHOUT ANY WARRANTY; without even the implied warranty of
\r
338 ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
\r
339 ;; GNU General Public License for more details.
\r
341 ;; You should have received a copy of the GNU General Public License
\r
342 ;; along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>.
\r
346 ;; This is a library for parsing and generating JSON (JavaScript Object
\r
349 ;; Learn all about JSON here: <URL:http://json.org/>.
\r
351 ;; The user-serviceable entry points for the parser are the functions
\r
352 ;; `json-read' and `json-read-from-string'. The encoder has a single
\r
353 ;; entry point, `json-encode'.
\r
355 ;; Since there are several natural representations of key-value pair
\r
356 ;; mappings in elisp (alist, plist, hash-table), `json-read' allows you
\r
357 ;; to specify which you'd prefer (see `json-object-type' and
\r
358 ;; `json-array-type').
\r
360 ;; Similarly, since `false' and `null' are distinct in JSON, you can
\r
361 ;; distinguish them by binding `json-false' and `json-null' as desired.
\r
365 ;; 2011-07-20 - Optimized by Austin Clements <aclements@csail.mit.edu>.
\r
366 ;; 2006-03-11 - Initial version.
\r
367 ;; 2006-03-13 - Added JSON generation in addition to parsing. Various
\r
368 ;; other cleanups, bugfixes, and improvements.
\r
369 ;; 2006-12-29 - XEmacs support, from Aidan Kehoe <kehoea@parhasard.net>.
\r
370 ;; 2008-02-21 - Installed in GNU Emacs.
\r
374 (eval-when-compile (require 'cl))
\r
376 ;; Compatibility code
\r
378 (defalias 'json-encode-char0 'encode-char)
\r
379 (defalias 'json-decode-char0 'decode-char)
\r
384 (defvar json-object-type 'alist
\r
385 "Type to convert JSON objects to.
\r
386 Must be one of `alist', `plist', or `hash-table'. Consider let-binding
\r
387 this around your call to `json-read' instead of `setq'ing it.")
\r
389 (defvar json-array-type 'vector
\r
390 "Type to convert JSON arrays to.
\r
391 Must be one of `vector' or `list'. Consider let-binding this around
\r
392 your call to `json-read' instead of `setq'ing it.")
\r
394 (defvar json-key-type nil
\r
395 "Type to convert JSON keys to.
\r
396 Must be one of `string', `symbol', `keyword', or nil.
\r
398 If nil, `json-read' will guess the type based on the value of
\r
399 `json-object-type':
\r
401 If `json-object-type' is: nil will be interpreted as:
\r
402 `hash-table' `string'
\r
406 Note that values other than `string' might behave strangely for
\r
407 Sufficiently Weird keys. Consider let-binding this around your call to
\r
408 `json-read' instead of `setq'ing it.")
\r
410 (defvar json-false :json-false
\r
411 "Value to use when reading JSON `false'.
\r
412 If this has the same value as `json-null', you might not be able to tell
\r
413 the difference between `false' and `null'. Consider let-binding this
\r
414 around your call to `json-read' instead of `setq'ing it.")
\r
416 (defvar json-null nil
\r
417 "Value to use when reading JSON `null'.
\r
418 If this has the same value as `json-false', you might not be able to
\r
419 tell the difference between `false' and `null'. Consider let-binding
\r
420 this around your call to `json-read' instead of `setq'ing it.")
\r
427 (defun json-join (strings separator)
\r
428 "Join STRINGS with SEPARATOR."
\r
429 (mapconcat 'identity strings separator))
\r
431 (defun json-alist-p (list)
\r
432 "Non-null if and only if LIST is an alist."
\r
434 (and (consp (car list))
\r
435 (json-alist-p (cdr list)))))
\r
437 (defun json-plist-p (list)
\r
438 "Non-null if and only if LIST is a plist."
\r
440 (and (keywordp (car list))
\r
442 (json-plist-p (cddr list)))))
\r
444 ;; Reader utilities
\r
446 ;; (defsubst json-advance (&optional n)
\r
447 ;; "Skip past the following N characters."
\r
448 ;; (forward-char n))
\r
450 (defalias 'json-advance 'forward-char)
\r
452 ;; (defsubst json-peek ()
\r
453 ;; "Return the character at point."
\r
454 ;; (let ((char (char-after (point))))
\r
455 ;; (or char :json-eof)))
\r
457 (defsubst json-peek ()
\r
458 "Return the character at point."
\r
459 (or (char-after) :json-eof))
\r
461 (defsubst json-pop ()
\r
462 "Advance past the character at point, returning it."
\r
463 (let ((char (json-peek)))
\r
464 (if (eq char :json-eof)
\r
465 (signal 'end-of-file nil)
\r
469 ;; (defun json-skip-whitespace ()
\r
470 ;; "Skip past the whitespace at point."
\r
471 ;; (skip-chars-forward "\t\r\n\f\b "))
\r
473 (defsubst json-skip-whitespace ()
\r
474 "Skip past the whitespace at point."
\r
475 (skip-chars-forward "\t\r\n\f\b "))
\r
480 ;; Error conditions
\r
482 (put 'json-error 'error-message "Unknown JSON error")
\r
483 (put 'json-error 'error-conditions '(json-error error))
\r
485 (put 'json-readtable-error 'error-message "JSON readtable error")
\r
486 (put 'json-readtable-error 'error-conditions
\r
487 '(json-readtable-error json-error error))
\r
489 (put 'json-unknown-keyword 'error-message "Unrecognized keyword")
\r
490 (put 'json-unknown-keyword 'error-conditions
\r
491 '(json-unknown-keyword json-error error))
\r
493 (put 'json-number-format 'error-message "Invalid number format")
\r
494 (put 'json-number-format 'error-conditions
\r
495 '(json-number-format json-error error))
\r
497 (put 'json-string-escape 'error-message "Bad unicode escape")
\r
498 (put 'json-string-escape 'error-conditions
\r
499 '(json-string-escape json-error error))
\r
501 (put 'json-string-format 'error-message "Bad string format")
\r
502 (put 'json-string-format 'error-conditions
\r
503 '(json-string-format json-error error))
\r
505 (put 'json-object-format 'error-message "Bad JSON object")
\r
506 (put 'json-object-format 'error-conditions
\r
507 '(json-object-format json-error error))
\r
514 (defvar json-keywords '("true" "false" "null")
\r
515 "List of JSON keywords.")
\r
519 (defun json-read-keyword (keyword)
\r
520 "Read a JSON keyword at point.
\r
521 KEYWORD is the keyword expected."
\r
522 (unless (member keyword json-keywords)
\r
523 (signal 'json-unknown-keyword (list keyword)))
\r
524 (mapc (lambda (char)
\r
525 (unless (char-equal char (json-peek))
\r
526 (signal 'json-unknown-keyword
\r
527 (list (save-excursion
\r
529 (thing-at-point 'word)))))
\r
532 (unless (looking-at "\\(\\s-\\|[],}]\\|$\\)")
\r
533 (signal 'json-unknown-keyword
\r
534 (list (save-excursion
\r
536 (thing-at-point 'word)))))
\r
537 (cond ((string-equal keyword "true") t)
\r
538 ((string-equal keyword "false") json-false)
\r
539 ((string-equal keyword "null") json-null)))
\r
541 ;; Keyword encoding
\r
543 (defun json-encode-keyword (keyword)
\r
544 "Encode KEYWORD as a JSON value."
\r
545 (cond ((eq keyword t) "true")
\r
546 ((eq keyword json-false) "false")
\r
547 ((eq keyword json-null) "null")))
\r
553 ;; (defun json-read-number (&optional sign)
\r
554 ;; "Read the JSON number following point.
\r
555 ;; The optional SIGN argument is for internal use.
\r
557 ;; N.B.: Only numbers which can fit in Emacs Lisp's native number
\r
558 ;; representation will be parsed correctly."
\r
559 ;; ;; If SIGN is non-nil, the number is explicitly signed.
\r
560 ;; (let ((number-regexp
\r
561 ;; "\\([0-9]+\\)?\\(\\.[0-9]+\\)?\\([Ee][+-]?[0-9]+\\)?"))
\r
562 ;; (cond ((and (null sign) (char-equal (json-peek) ?-))
\r
564 ;; (- (json-read-number t)))
\r
565 ;; ((and (null sign) (char-equal (json-peek) ?+))
\r
567 ;; (json-read-number t))
\r
568 ;; ((and (looking-at number-regexp)
\r
569 ;; (or (match-beginning 1)
\r
570 ;; (match-beginning 2)))
\r
571 ;; (goto-char (match-end 0))
\r
572 ;; (string-to-number (match-string 0)))
\r
573 ;; (t (signal 'json-number-format (list (point)))))))
\r
575 (defun json-read-number ()
\r
576 "Read the JSON number following point.
\r
578 N.B.: Only numbers which can fit in Emacs Lisp's native number
\r
579 representation will be parsed correctly."
\r
580 ;; This regexp requires one character of backtrack in the common case
\r
581 ;; of a whole number, but is slightly faster than a more explicit
\r
582 ;; regexp like "\\([0-9]+\\)?\\(\\.[0-9]+\\)?"
\r
583 (if (looking-at "[-+]?[0-9]*[.0-9][0-9]*\\([Ee][+-]?[0-9]+\\)?")
\r
585 (goto-char (match-end 0))
\r
586 (string-to-number (match-string 0)))
\r
587 (signal 'json-number-format (list (point)))))
\r
591 (defun json-encode-number (number)
\r
592 "Return a JSON representation of NUMBER."
\r
593 (format "%s" number))
\r
597 (defvar json-special-chars
\r
606 "Characters which are escaped in JSON, with their elisp counterparts.")
\r
610 (defun json-read-escaped-char ()
\r
611 "Read the JSON string escaped character at point."
\r
612 ;; Skip over the '\'
\r
614 (let* ((char (json-pop))
\r
615 (special (assq char json-special-chars)))
\r
617 (special (cdr special))
\r
618 ((not (eq char ?u)) char)
\r
619 ((looking-at "[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]")
\r
620 (let ((hex (match-string 0)))
\r
622 (json-decode-char0 'ucs (string-to-number hex 16))))
\r
624 (signal 'json-string-escape (list (point)))))))
\r
626 ;; (defun json-read-string ()
\r
627 ;; "Read the JSON string at point."
\r
628 ;; (unless (char-equal (json-peek) ?\")
\r
629 ;; (signal 'json-string-format (list "doesn't start with '\"'!")))
\r
630 ;; ;; Skip over the '"'
\r
632 ;; (let ((characters '())
\r
633 ;; (char (json-peek)))
\r
634 ;; (while (not (char-equal char ?\"))
\r
635 ;; (push (if (char-equal char ?\\)
\r
636 ;; (json-read-escaped-char)
\r
639 ;; (setq char (json-peek)))
\r
640 ;; ;; Skip over the '"'
\r
643 ;; (apply 'string (nreverse characters))
\r
647 (defun json-read-string ()
\r
648 "Read the JSON string at point."
\r
649 ;; (unless (char-equal (json-peek) ?\")
\r
650 ;; (signal 'json-string-format (list "doesn't start with '\"'!")))
\r
651 ;; Skip over the '"'
\r
653 (let ((parts '()) (more t))
\r
655 (let ((start (point)))
\r
656 (when (> (skip-chars-forward "^\\\\\"") 0)
\r
657 (push (buffer-substring-no-properties start (point)) parts)))
\r
659 (let ((char (char-after)))
\r
660 (cond ((eq char ?\") (json-advance) (setq more nil))
\r
661 ((eq char ?\\) (push (string (json-read-escaped-char)) parts))
\r
662 (t (error "XXX Unterminated string")))))
\r
663 ;; (let ((char (json-peek)))
\r
665 ;; ((?\") (json-advance) (setq done t))
\r
666 ;; ((?\\) (push (string (json-read-escaped-char)) parts))
\r
667 ;; (t (error "XXX Unterminated string")))))
\r
670 (apply 'concat (nreverse parts))
\r
676 (defun json-encode-char (char)
\r
677 "Encode CHAR as a JSON string."
\r
678 (setq char (json-encode-char0 char 'ucs))
\r
679 (let ((control-char (car (rassoc char json-special-chars))))
\r
681 ;; Special JSON character (\n, \r, etc.)
\r
683 (format "\\%c" control-char))
\r
684 ;; ASCIIish printable character
\r
685 ((and (> char 31) (< char 161))
\r
686 (format "%c" char))
\r
687 ;; Fallback: UCS code point in \uNNNN form
\r
689 (format "\\u%04x" char)))))
\r
691 (defun json-encode-string (string)
\r
692 "Return a JSON representation of STRING."
\r
693 (format "\"%s\"" (mapconcat 'json-encode-char string "")))
\r
697 (defun json-new-object ()
\r
698 "Create a new Elisp object corresponding to a JSON object.
\r
699 Please see the documentation of `json-object-type'."
\r
700 (cond ((eq json-object-type 'hash-table)
\r
701 (make-hash-table :test 'equal))
\r
705 (defun json-add-to-object (object key value)
\r
706 "Add a new KEY -> VALUE association to OBJECT.
\r
707 Returns the updated object, which you should save, e.g.:
\r
708 (setq obj (json-add-to-object obj \"foo\" \"bar\"))
\r
709 Please see the documentation of `json-object-type' and `json-key-type'."
\r
710 (let ((json-key-type
\r
711 (if (eq json-key-type nil)
\r
712 (cdr (assq json-object-type '((hash-table . string)
\r
714 (plist . keyword))))
\r
717 (cond ((eq json-key-type 'string)
\r
719 ((eq json-key-type 'symbol)
\r
721 ((eq json-key-type 'keyword)
\r
722 (intern (concat ":" key)))))
\r
723 (cond ((eq json-object-type 'hash-table)
\r
724 (puthash key value object)
\r
726 ((eq json-object-type 'alist)
\r
727 (cons (cons key value) object))
\r
728 ((eq json-object-type 'plist)
\r
729 (cons key (cons value object))))))
\r
731 ;; JSON object parsing
\r
733 ;; (defun json-read-object ()
\r
734 ;; "Read the JSON object at point."
\r
735 ;; ;; Skip over the "{"
\r
737 ;; (json-skip-whitespace)
\r
738 ;; ;; read key/value pairs until "}"
\r
739 ;; (let ((elements (json-new-object))
\r
741 ;; (while (not (char-equal (json-peek) ?}))
\r
742 ;; (json-skip-whitespace)
\r
743 ;; (setq key (json-read-string))
\r
744 ;; (json-skip-whitespace)
\r
745 ;; (if (char-equal (json-peek) ?:)
\r
747 ;; (signal 'json-object-format (list ":" (json-peek))))
\r
748 ;; (setq value (json-read))
\r
749 ;; (setq elements (json-add-to-object elements key value))
\r
750 ;; (json-skip-whitespace)
\r
751 ;; (unless (char-equal (json-peek) ?})
\r
752 ;; (if (char-equal (json-peek) ?,)
\r
754 ;; (signal 'json-object-format (list "," (json-peek))))))
\r
755 ;; ;; Skip over the "}"
\r
760 (defun json-read-object ()
\r
761 "Read the JSON object at point."
\r
762 ;; Skip over the "{"
\r
764 (json-skip-whitespace)
\r
765 ;; read key/value pairs until "}"
\r
766 (let ((elements (json-new-object))
\r
767 key value (more t))
\r
768 (unless (eq (char-after) ?})
\r
769 ;; (while (not (eq (char-after) ?}))
\r
771 (unless (eq (char-after) ?\")
\r
772 (json-skip-whitespace)
\r
773 (unless (eq (char-after) ?\")
\r
774 (signal 'json-string-format (list "doesn't start with '\"'!"))))
\r
775 (setq key (json-read-string))
\r
776 ;; Makes a small but surprising difference, adds up if done
\r
778 (if (eq (char-after) ?:)
\r
780 (if (progn (json-skip-whitespace) (eq (char-after) ?:))
\r
782 (signal 'json-object-format (list ":" (json-peek)))))
\r
783 (setq value (json-read))
\r
784 (setq elements (json-add-to-object elements key value))
\r
785 ;; Order matters a little
\r
786 (cond ((eq (char-after) ?,) (json-advance))
\r
787 ((eq (char-after) ?}) (setq more nil))
\r
789 (json-skip-whitespace)
\r
790 (eq (char-after) ?,)) (json-advance))
\r
791 ((eq (char-after) ?}) (setq more nil))
\r
792 (t (signal 'json-object-format (list "," (json-peek)))))))
\r
793 ;; (unless (char-equal (json-peek) ?})
\r
794 ;; (if (char-equal (json-peek) ?,)
\r
796 ;; (signal 'json-object-format (list "," (json-peek))))))
\r
797 ;; Skip over the "}"
\r
801 ;; Hash table encoding
\r
803 (defun json-encode-hash-table (hash-table)
\r
804 "Return a JSON representation of HASH-TABLE."
\r
810 (push (format "%s:%s"
\r
818 ;; List encoding (including alists and plists)
\r
820 (defun json-encode-alist (alist)
\r
821 "Return a JSON representation of ALIST."
\r
823 (json-join (mapcar (lambda (cons)
\r
825 (json-encode (car cons))
\r
826 (json-encode (cdr cons))))
\r
830 (defun json-encode-plist (plist)
\r
831 "Return a JSON representation of PLIST."
\r
834 (push (concat (json-encode (car plist))
\r
836 (json-encode (cadr plist)))
\r
838 (setq plist (cddr plist)))
\r
839 (concat "{" (json-join (nreverse result) ", ") "}")))
\r
841 (defun json-encode-list (list)
\r
842 "Return a JSON representation of LIST.
\r
843 Tries to DWIM: simple lists become JSON arrays, while alists and plists
\r
844 become JSON objects."
\r
845 (cond ((null list) "null")
\r
846 ((json-alist-p list) (json-encode-alist list))
\r
847 ((json-plist-p list) (json-encode-plist list))
\r
848 ((listp list) (json-encode-array list))
\r
850 (signal 'json-error (list list)))))
\r
856 ;; (defun json-read-array ()
\r
857 ;; "Read the JSON array at point."
\r
858 ;; ;; Skip over the "["
\r
860 ;; (json-skip-whitespace)
\r
861 ;; ;; read values until "]"
\r
863 ;; (while (not (char-equal (json-peek) ?\]))
\r
864 ;; (push (json-read) elements)
\r
865 ;; (json-skip-whitespace)
\r
866 ;; (unless (char-equal (json-peek) ?\])
\r
867 ;; (if (char-equal (json-peek) ?,)
\r
869 ;; (signal 'json-error (list 'bleah)))))
\r
870 ;; ;; Skip over the "]"
\r
872 ;; (apply json-array-type (nreverse elements))))
\r
874 (defun json-read-array ()
\r
875 "Read the JSON array at point."
\r
876 ;; Skip over the "["
\r
878 (json-skip-whitespace)
\r
879 ;; read values until "]"
\r
880 (let* (elements (more t))
\r
881 (unless (eq (char-after) ?\])
\r
883 ;; (while (not (char-equal (json-peek) ?\]))
\r
884 (push (json-read) elements)
\r
886 ;; (setq tail (setcdr tail (cons (json-read) nil)))
\r
888 ;; (json-skip-whitespace)
\r
889 (cond ((eq (char-after) ?,) (json-advance))
\r
890 ((eq (char-after) ?\]) (setq more nil))
\r
892 (json-skip-whitespace)
\r
893 (eq (char-after) ?,)) (json-advance))
\r
894 ((eq (char-after) ?\]) (setq more nil))
\r
895 (t (signal 'json-error (list 'bleah))))))
\r
896 ;; (unless (char-equal (json-peek) ?\])
\r
897 ;; (if (char-equal (json-peek) ?,)
\r
899 ;; (signal 'json-error (list 'bleah))))))
\r
900 ;; Skip over the "]"
\r
903 (if (eq json-array-type 'list)
\r
904 (nreverse elements)
\r
905 (apply json-array-type (nreverse elements)))))
\r
909 (defun json-encode-array (array)
\r
910 "Return a JSON representation of ARRAY."
\r
911 (concat "[" (mapconcat 'json-encode array ", ") "]"))
\r
918 ;; (defvar json-readtable
\r
920 ;; '((?t json-read-keyword "true")
\r
921 ;; (?f json-read-keyword "false")
\r
922 ;; (?n json-read-keyword "null")
\r
923 ;; (?{ json-read-object)
\r
924 ;; (?\[ json-read-array)
\r
925 ;; (?\" json-read-string))))
\r
926 ;; (mapc (lambda (char)
\r
927 ;; (push (list char 'json-read-number) table))
\r
928 ;; '(?- ?+ ?. ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
\r
930 ;; "Readtable for JSON reader.")
\r
932 ;; (defun json-read ()
\r
933 ;; "Parse and return the JSON object following point.
\r
934 ;; Advances point just past JSON object."
\r
935 ;; (json-skip-whitespace)
\r
936 ;; (let ((char (json-peek)))
\r
937 ;; (if (not (eq char :json-eof))
\r
938 ;; (let ((record (cdr (assq char json-readtable))))
\r
939 ;; (if (functionp (car record))
\r
940 ;; (apply (car record) (cdr record))
\r
941 ;; (signal 'json-readtable-error record)))
\r
942 ;; (signal 'end-of-file nil))))
\r
944 (defvar my-json-readtable
\r
945 (let ((table (make-char-table nil)))
\r
946 (aset table ?t '(json-read-keyword "true"))
\r
947 (aset table ?f '(json-read-keyword "false"))
\r
948 (aset table ?n '(json-read-keyword "null"))
\r
949 (aset table ?{ '(json-read-object))
\r
950 (aset table ?\[ '(json-read-array))
\r
951 (aset table ?\" '(json-read-string))
\r
952 (mapc (lambda (char)
\r
953 (aset table char '(json-read-number)))
\r
954 '(?- ?+ ?. ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
\r
956 "Readtable for JSON reader.")
\r
958 ;; Char-table matters a bit; (if (null ..)) matters more
\r
959 ;; (defun json-read ()
\r
960 ;; "Parse and return the JSON object following point.
\r
961 ;; Advances point just past JSON object."
\r
962 ;; (json-skip-whitespace)
\r
963 ;; (let ((char (json-peek)))
\r
964 ;; (if (not (eq char :json-eof))
\r
965 ;; (let ((record (aref my-json-readtable char)))
\r
966 ;; (if (null (car record))
\r
967 ;; (signal 'json-readtable-error record)
\r
968 ;; (apply (car record) (cdr record))))
\r
969 ;; (signal 'end-of-file nil))))
\r
971 ;; Makes no difference or slower, probably because there's usually whitespace
\r
972 ;; (defun json-read ()
\r
973 ;; "Parse and return the JSON object following point.
\r
974 ;; Advances point just past JSON object."
\r
975 ;; (let ((record (and (char-after) (aref my-json-readtable (char-after)))))
\r
976 ;; (when (null record)
\r
977 ;; (json-skip-whitespace)
\r
979 ;; (signal 'end-of-file nil))
\r
980 ;; (setq record (aref my-json-readtable (char-after)))
\r
981 ;; (when (null record)
\r
982 ;; (signal 'json-readtable-error record)))
\r
983 ;; (apply (car record) (cdr record))))
\r
985 ;; Makes a difference
\r
986 (defun json-read ()
\r
987 "Parse and return the JSON object following point.
\r
988 Advances point just past JSON object."
\r
989 (json-skip-whitespace)
\r
991 (let ((record (aref my-json-readtable (char-after))))
\r
993 (apply (car record) (cdr record))
\r
994 (signal 'json-readtable-error record)))
\r
995 (signal 'end-of-file nil)))
\r
997 ;; Syntactic sugar for the reader
\r
999 (defun json-read-from-string (string)
\r
1000 "Read the JSON object contained in STRING and return it."
\r
1003 (goto-char (point-min))
\r
1006 (defun json-read-file (file)
\r
1007 "Read the first JSON object contained in FILE and return it."
\r
1009 (insert-file-contents file)
\r
1010 (goto-char (point-min))
\r
1018 (defun json-encode (object)
\r
1019 "Return a JSON representation of OBJECT as a string."
\r
1020 (cond ((memq object (list t json-null json-false))
\r
1021 (json-encode-keyword object))
\r
1022 ((stringp object) (json-encode-string object))
\r
1023 ((keywordp object) (json-encode-string
\r
1024 (substring (symbol-name object) 1)))
\r
1025 ((symbolp object) (json-encode-string
\r
1026 (symbol-name object)))
\r
1027 ((numberp object) (json-encode-number object))
\r
1028 ((arrayp object) (json-encode-array object))
\r
1029 ((hash-table-p object) (json-encode-hash-table object))
\r
1030 ((listp object) (json-encode-list object))
\r
1031 (t (signal 'json-error (list object)))))
\r
1035 ;; arch-tag: 15f6e4c8-b831-4172-8749-bbc680c50ea1
\r
1036 ;;; json.el ends here
\r
1038 --/04w6evG8XlLl3ft--
\r