--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id EA757431FBC\r
+ for <notmuch@notmuchmail.org>; Sun, 1 Jul 2012 17:12:43 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id MxFIJeuIzkrK for <notmuch@notmuchmail.org>;\r
+ Sun, 1 Jul 2012 17:12:42 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-3.mit.edu (DMZ-MAILSEC-SCANNER-3.MIT.EDU\r
+ [18.9.25.14])\r
+ by olra.theworths.org (Postfix) with ESMTP id 32899431FAF\r
+ for <notmuch@notmuchmail.org>; Sun, 1 Jul 2012 17:12:42 -0700 (PDT)\r
+X-AuditID: 1209190e-b7fb56d0000008b2-61-4ff0e779327b\r
+Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
+ by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 0D.6B.02226.977E0FF4; Sun, 1 Jul 2012 20:12:41 -0400 (EDT)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id q620CeXv017423; \r
+ Sun, 1 Jul 2012 20:12:41 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q620CaLp024059\r
+ (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+ Sun, 1 Jul 2012 20:12:39 -0400 (EDT)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1SlUFo-0007LF-50; Sun, 01 Jul 2012 20:12:36 -0400\r
+Date: Sun, 1 Jul 2012 20:12:35 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Tomi Ollila <tomi.ollila@iki.fi>\r
+Subject: Re: [PATCH] cli: notmuch-show with framing newlines between threads\r
+ in JSON.\r
+Message-ID: <20120702001215.GC6220@mit.edu>\r
+References: <1341041024-5342-1-git-send-email-markwalters1009@gmail.com>\r
+ <m24npr8701.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <m24npr8701.fsf@guru.guru-group.fi>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFprOKsWRmVeSWpSXmKPExsUixCmqrVv5/IO/wZZmRYvVc3ksrt+cyWzx\r
+ ZuU8Vgdmj52z7rJ7HP66kMXj2apbzAHMUVw2Kak5mWWpRfp2CVwZF7c+YS64IFtx6ddnpgbG\r
+ TeJdjJwcEgImEnvW/mKGsMUkLtxbz9bFyMUhJLCPUaJ1Xyc7hLOeUWLuvglMEM4JJomJx4+z\r
+ grQICSxhlFi3OqqLkYODRUBF4sz0BJAwm4CGxLb9yxlBbBGg8IO29WDlzAKuEjMu7GICsYUF\r
+ wiUOb9rCBmLzCmhL3P3fywYxMk9i4renUHFBiZMzn7BA9GpJ3Pj3kglkFbOAtMTyfxwgYU4B\r
+ A4kt71aDjRQFWjXl5Da2CYxCs5B0z0LSPQuhewEj8ypG2ZTcKt3cxMyc4tRk3eLkxLy81CJd\r
+ Y73czBK91JTSTYzgMJfk28H49aDSIUYBDkYlHl7l2x/8hVgTy4orcw8xSnIwKYnyHn8MFOJL\r
+ yk+pzEgszogvKs1JLT7EKMHBrCTC+/T4e38h3pTEyqrUonyYlDQHi5I475WUm/5CAumJJanZ\r
+ qakFqUUwWRkODiUJ3lnPgIYKFqWmp1akZeaUIKSZODhBhvMADT8KUsNbXJCYW5yZDpE/xajL\r
+ se7NkRuMQix5+XmpUuK8G0CKBECKMkrz4ObA0tMrRnGgt4R5J4FU8QBTG9ykV0BLmICWPF/9\r
+ DmRJSSJCSqqBcZLB+Wkf3wpyT9+o+yL+fEZoZuCWM8ap1ZHzpn9bzDPDZEHF7oNWlx1UxT/k\r
+ CO16V/CbY01/lZWv98nfk5RznAslctYszy5l16yv3vb+esTcnL1Fu2cc/ln4ScxrUpH52Ut1\r
+ gqopL2rWy3nFXq2ZsH7OcU7ei7KyuxxV07Kcvp8s63own8mdR4mlOCPRUIu5qDgRAEZ2JCAq AwAA\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 02 Jul 2012 00:12:44 -0000\r
+\r
+Quoth Tomi Ollila on Jul 02 at 1:13 am:\r
+> On Sat, Jun 30 2012, Mark Walters <markwalters1009@gmail.com> wrote:\r
+> \r
+> > Add newlines between complete threads to make asynchronous parsing\r
+> > of the JSON easier.\r
+> > ---\r
+> >\r
+> > notmuch-pick uses the JSON output of notmuch show but, in many cases,\r
+> > for many threads. This can take quite a long time when displaying a\r
+> > large number of messages (say 20 seconds for the 10,000 messages in\r
+> > the notmuch archive). Thus it is desirable to display results\r
+> > incrementally in the same way that search currently does.\r
+> >\r
+> > To make this easier this patch adds newlines between each toplevel\r
+> > thread. So the ouput becomes\r
+> >\r
+> > [\r
+> > thread1\r
+> > , thread2\r
+> > , thread3\r
+> > ...\r
+> > , last_thread\r
+> > ]\r
+> >\r
+> > Thus the parser can easily tell if it has enough data to do some more\r
+> > parsing.\r
+> >\r
+> > Obviously, this changes the JSON output. This should not break any\r
+> > consumer as the JSON parsers should not mind. However, it does break\r
+> > several tests. Obviously, I will fix these but I wanted to check if\r
+> > people were basically happy with the change first.\r
+> \r
+> To provide this feature rather than relying on newlines the parser should\r
+> use it's state to notice when one thread ends. \r
+> \r
+> Such a change could be used (privately) for human consumption -- allowing \r
+> free change of whitespace during inspection (in a debugging session or so).\r
+> Computer software should not rely (or suffer) from any additional\r
+> (or lack thereof) whitespace there is...\r
+> \r
+> ... or at least a really convicing argument for the chance needs to\r
+> be presented (before "restricting" the json output notmuch spits out).\r
+\r
+Given a JSON parser that only knows how to parse complete JSON\r
+expressions, it's potentially very inefficient to keep attempting to\r
+parse something when you don't know if it's complete. The newlines\r
+provide an in-band framing so the consumer knows when there's a\r
+complete object to be parsed.\r
+\r
+In effect, this defines a super-protocol of JSON that's compatible\r
+with standard JSON, but easy to incrementally parse.\r
+\r
+That said, just this weekend I implemented JSON-based search with\r
+incremental JSON parsing and I took a slightly different approach. I\r
+still put framing into the newlines of the search results, but rather\r
+than rely on it for correctness, the consumer uses it as an\r
+optimization that only hints that a complete JSON expression is\r
+probably available. If the expression turns out to be incomplete,\r
+that's okay.\r
+\r
+I considered building a fully-incremental JSON parser that never\r
+backtracks by more than a token, which would eliminate even the cost\r
+of reparsing, but if we do move to S-expressions (which I think we\r
+should), we want to let Emacs' C implementation do as much of the\r
+parsing as possible, and the only thing we can do with that is read a\r
+complete expression.\r
+\r
+> Btw: AFAIC (json-read) parses the whole json object (ignoring whitespace,\r
+> including newlines outside strings). So I quess notmuch-pick uses something\r
+> slightly different (probably using json.el subroutines)..\r
+> \r
+> Btw2: I'm very interested to see notmuch-pick in action -- I just don't\r
+> see this a way to do this particular support properly.\r
+> \r
+> Btw3: is search is ever going to use json we'll face the same problem -- \r
+> unless writing each line as a separate json object (and starting to use \r
+> s-expressions for speed)\r
+\r
+Done. I'll post the patches after a little more cleanup.\r
+\r
+> > Also, should devel/schemata be updated? It seems a little unclear as\r
+> > this is not really a "JSON" change as the JSON does not care about the\r
+> > newlines.\r
+> >\r
+> > Best wishes\r
+> \r
+> and best luck with your notmuch-pick work.\r
+> \r
+> >\r
+> > Mark\r
+> \r
+> Tomi\r
projects / notmuch-archives.git / commitdiff