From: Austin Clements Date: Mon, 2 Jul 2012 03:52:41 +0000 (+2000) Subject: Re: [PATCH] cli: notmuch-show with framing newlines between threads in JSON. X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=a5927bed7484f1097b5fbd5ea45d92e949158ca2;p=notmuch-archives.git Re: [PATCH] cli: notmuch-show with framing newlines between threads in JSON. --- diff --git a/59/e08b565eb2d97891bde2be9772ca8fceee8fbc b/59/e08b565eb2d97891bde2be9772ca8fceee8fbc new file mode 100644 index 000000000..354450990 --- /dev/null +++ b/59/e08b565eb2d97891bde2be9772ca8fceee8fbc @@ -0,0 +1,154 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 4BF5A431FBC + for ; Sun, 1 Jul 2012 20:52:48 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id 5zw4IQ-ybvPT for ; + Sun, 1 Jul 2012 20:52:47 -0700 (PDT) +Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU + [18.9.25.15]) + by olra.theworths.org (Postfix) with ESMTP id 953E3431FAF + for ; Sun, 1 Jul 2012 20:52:47 -0700 (PDT) +X-AuditID: 1209190f-b7f306d0000008b4-a8-4ff11b0d0601 +Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) + by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP + id C5.01.02228.D0B11FF4; Sun, 1 Jul 2012 23:52:46 -0400 (EDT) +Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) + by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id q623qjul011592; + Sun, 1 Jul 2012 23:52:45 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q623qgRi023541 + (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); + Sun, 1 Jul 2012 23:52:44 -0400 (EDT) +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77) + (envelope-from ) + id 1SlXgn-0007kM-VK; Sun, 01 Jul 2012 23:52:42 -0400 +Date: Sun, 1 Jul 2012 23:52:41 -0400 +From: Austin Clements +To: Tomi Ollila +Subject: Re: [PATCH] cli: notmuch-show with framing newlines between threads + in JSON. +Message-ID: <20120702035241.GD6220@mit.edu> +References: <1341041024-5342-1-git-send-email-markwalters1009@gmail.com> + <20120702001215.GC6220@mit.edu> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <20120702001215.GC6220@mit.edu> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFprAKsWRmVeSWpSXmKPExsUixCmqrMsn/dHfYONdRovrN2cyW7xZOY/V + gcnj8NeFLB7PVt1iDmCK4rJJSc3JLEst0rdL4Mr4t0KioFW6YmrfJtYGxmmiXYycHBICJhJz + Nl5ngrDFJC7cW8/WxcjFISSwj1Fi8ckFrBDOekaJ04+nMUI4J5gkZr/fBuUsYZTYfvoDO0g/ + i4CKxPnG32Cz2AQ0JLbtX84IYosAxR+0rWcFsZkFpCW+/W4GqxEWCJc4vGkLG4jNK6AtsW3u + exaIoRMYJSa9nMkKkRCUODnzCQtEs5bEjX8vgZo5wAYt/8cBEuYU0JHY3rwR7AZRoF1TTm5j + m8AoNAtJ9ywk3bMQuhcwMq9ilE3JrdLNTczMKU5N1i1OTszLSy3SNdHLzSzRS00p3cQIDmxJ + /h2M3w4qHWIU4GBU4uFVuv3BX4g1say4MvcQoyQHk5Io70qJj/5CfEn5KZUZicUZ8UWlOanF + hxglOJiVRHifHn/vL8SbklhZlVqUD5OS5mBREue9mnLTX0ggPbEkNTs1tSC1CCYrw8GhJMEb + LQU0VLAoNT21Ii0zpwQhzcTBCTKcB2j4apAa3uKCxNzizHSI/ClGXY51b47cYBRiycvPS5US + 550FUiQAUpRRmgc3B5aQXjGKA70lzOsBUsUDTGZwk14BLWECWvJ89TuQJSWJCCmpBkbWoCOz + mc4khVzg6nbuYPLmejA3pJrbyS/WebJQ46mpl9RWfA7avVTQOl40PFD54VOD0Ef7k6TWrfvz + Qu2+bdZ/1mePQySnB35/tOlK5J17Tv5Ff03fWbS1iSbNCZtu6HhCfk6XXdIaD7XzffeTHh1q + Zip82GR20nBbl9SnfCE1gdKZRSvio5VYijMSDbWYi4oTARR0Lx8jAwAA +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 02 Jul 2012 03:52:48 -0000 + +Quoth myself on Jul 01 at 8:12 pm: +> Quoth Tomi Ollila on Jul 02 at 1:13 am: +> > On Sat, Jun 30 2012, Mark Walters wrote: +> > +> > > Add newlines between complete threads to make asynchronous parsing +> > > of the JSON easier. +> > > --- +> > > +> > > notmuch-pick uses the JSON output of notmuch show but, in many cases, +> > > for many threads. This can take quite a long time when displaying a +> > > large number of messages (say 20 seconds for the 10,000 messages in +> > > the notmuch archive). Thus it is desirable to display results +> > > incrementally in the same way that search currently does. +> > > +> > > To make this easier this patch adds newlines between each toplevel +> > > thread. So the ouput becomes +> > > +> > > [ +> > > thread1 +> > > , thread2 +> > > , thread3 +> > > ... +> > > , last_thread +> > > ] +> > > +> > > Thus the parser can easily tell if it has enough data to do some more +> > > parsing. +> > > +> > > Obviously, this changes the JSON output. This should not break any +> > > consumer as the JSON parsers should not mind. However, it does break +> > > several tests. Obviously, I will fix these but I wanted to check if +> > > people were basically happy with the change first. +> > +> > To provide this feature rather than relying on newlines the parser should +> > use it's state to notice when one thread ends. +> > +> > Such a change could be used (privately) for human consumption -- allowing +> > free change of whitespace during inspection (in a debugging session or so). +> > Computer software should not rely (or suffer) from any additional +> > (or lack thereof) whitespace there is... +> > +> > ... or at least a really convicing argument for the chance needs to +> > be presented (before "restricting" the json output notmuch spits out). +> +> Given a JSON parser that only knows how to parse complete JSON +> expressions, it's potentially very inefficient to keep attempting to +> parse something when you don't know if it's complete. The newlines +> provide an in-band framing so the consumer knows when there's a +> complete object to be parsed. +> +> In effect, this defines a super-protocol of JSON that's compatible +> with standard JSON, but easy to incrementally parse. +> +> That said, just this weekend I implemented JSON-based search with +> incremental JSON parsing and I took a slightly different approach. I +> still put framing into the newlines of the search results, but rather +> than rely on it for correctness, the consumer uses it as an +> optimization that only hints that a complete JSON expression is +> probably available. If the expression turns out to be incomplete, +> that's okay. +> +> I considered building a fully-incremental JSON parser that never +> backtracks by more than a token, which would eliminate even the cost +> of reparsing, but if we do move to S-expressions (which I think we +> should), we want to let Emacs' C implementation do as much of the +> parsing as possible, and the only thing we can do with that is read a +> complete expression. + +Actually, I take that back. While we can't do fast incremental +S-expression parsing, `parse-partial-sexp' can tell us incrementally +(and probably very quickly) *if* there's a complete expression ready +to parse, so we could avoid calling into the parser at all unless it +would succeed. + +I'll try this out in my incremental JSON parser and see how well it +works.