From 3dd7ca29e885b65dfc810d4662c086cdb3380f44 Mon Sep 17 00:00:00 2001 From: Mark Walters Date: Sun, 1 Jul 2012 23:43:57 +0100 Subject: [PATCH] Re: [PATCH] cli: notmuch-show with framing newlines between threads in JSON. --- 40/118e1904a8b284a2c5baf3f532a003ec4d000e | 188 ++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 40/118e1904a8b284a2c5baf3f532a003ec4d000e diff --git a/40/118e1904a8b284a2c5baf3f532a003ec4d000e b/40/118e1904a8b284a2c5baf3f532a003ec4d000e new file mode 100644 index 000000000..0bd3826a2 --- /dev/null +++ b/40/118e1904a8b284a2c5baf3f532a003ec4d000e @@ -0,0 +1,188 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 37189431FBC + for ; Sun, 1 Jul 2012 15:44:10 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -1.098 +X-Spam-Level: +X-Spam-Status: No, score=-1.098 tagged_above=-999 required=5 + tests=[DKIM_ADSP_CUSTOM_MED=0.001, FREEMAIL_FROM=0.001, + NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id s6NBfQvVZE0z for ; + Sun, 1 Jul 2012 15:44:09 -0700 (PDT) +Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 1A4E2431FAF + for ; Sun, 1 Jul 2012 15:44:09 -0700 (PDT) +Received: from smtp.qmul.ac.uk ([138.37.6.40]) + by mail2.qmul.ac.uk with esmtp (Exim 4.71) + (envelope-from ) + id 1SlSs6-0005fW-2W; Sun, 01 Jul 2012 23:44:07 +0100 +Received: from 94-192-233-223.zone6.bethere.co.uk ([94.192.233.223] + helo=localhost) + by smtp.qmul.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) + (envelope-from ) + id 1SlSs5-0001Uq-MY; Sun, 01 Jul 2012 23:44:01 +0100 +From: Mark Walters +To: Tomi Ollila , notmuch@notmuchmail.org +Subject: Re: [PATCH] cli: notmuch-show with framing newlines between threads + in JSON. +In-Reply-To: +References: <1341041024-5342-1-git-send-email-markwalters1009@gmail.com> + +User-Agent: Notmuch/0.13.2+70~gb6a56e7 (http://notmuchmail.org) Emacs/23.4.1 + (x86_64-pc-linux-gnu) +Date: Sun, 01 Jul 2012 23:43:57 +0100 +Message-ID: <8762a72jbm.fsf@qmul.ac.uk> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-Sender-Host-Address: 94.192.233.223 +X-QM-SPAM-Info: Sender has good ham record. :) +X-QM-Body-MD5: ca11c5bc5f73be30bedae6124641d972 (of first 20000 bytes) +X-SpamAssassin-Score: -1.8 +X-SpamAssassin-SpamBar: - +X-SpamAssassin-Report: The QM spam filters have analysed this message to + determine if it is + spam. We require at least 5.0 points to mark a message as spam. + This message scored -1.8 points. + Summary of the scoring: + * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, + * medium trust + * [138.37.6.40 listed in list.dnswl.org] + * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail + provider * (markwalters1009[at]gmail.com) + * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay + * domain + * 0.5 AWL AWL: From: address is in the auto white-list +X-QM-Scan-Virus: ClamAV says the message is clean +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 01 Jul 2012 22:44:10 -0000 + +On Sun, 01 Jul 2012, Tomi Ollila wrote: +> On Sat, Jun 30 2012, Mark Walters wrote: +> +>> Add newlines between complete threads to make asynchronous parsing +>> of the JSON easier. +>> --- +>> +>> notmuch-pick uses the JSON output of notmuch show but, in many cases, +>> for many threads. This can take quite a long time when displaying a +>> large number of messages (say 20 seconds for the 10,000 messages in +>> the notmuch archive). Thus it is desirable to display results +>> incrementally in the same way that search currently does. +>> +>> To make this easier this patch adds newlines between each toplevel +>> thread. So the ouput becomes +>> +>> [ +>> thread1 +>> , thread2 +>> , thread3 +>> ... +>> , last_thread +>> ] +>> +>> Thus the parser can easily tell if it has enough data to do some more +>> parsing. +>> +>> Obviously, this changes the JSON output. This should not break any +>> consumer as the JSON parsers should not mind. However, it does break +>> several tests. Obviously, I will fix these but I wanted to check if +>> people were basically happy with the change first. +> +> To provide this feature rather than relying on newlines the parser should +> use it's state to notice when one thread ends. +> +> Such a change could be used (privately) for human consumption -- allowing +> free change of whitespace during inspection (in a debugging session or so). +> Computer software should not rely (or suffer) from any additional +> (or lack thereof) whitespace there is... +> +> ... or at least a really convicing argument for the chance needs to +> be presented (before "restricting" the json output notmuch spits out). +> +> Btw: AFAIC (json-read) parses the whole json object (ignoring whitespace, +> including newlines outside strings). So I quess notmuch-pick uses something +> slightly different (probably using json.el subroutines).. + +I was following Austin's suggestion (on irc and +id:"20120214152114.GQ27039@mit.edu"). The idea is that each thread in +the JSON output is an entire JSON object. Thus pick skips the first [ +and the waits until there are two \n's in the incoming stream. Then it +knows that the complete first thread has been received and it parses +that with json-read as normal. The important thing is that it is trivial +to tell when a complete (and so parsable) JSON object has arrived. + +It seems to work, but I am definitely open to other approaches. + +> Btw2: I'm very interested to see notmuch-pick in action -- I just don't +> see this a way to do this particular support properly. +> +> Btw3: is search is ever going to use json we'll face the same problem -- +> unless writing each line as a separate json object (and starting to use +> s-expressions for speed) +> +>> Also, should devel/schemata be updated? It seems a little unclear as +>> this is not really a "JSON" change as the JSON does not care about the +>> newlines. +>> +>> Best wishes + + +> and best luck with your notmuch-pick work. + +Thanks! + +Mark + +>> +>> notmuch-show.c | 5 +++++ +>> 1 files changed, 5 insertions(+), 0 deletions(-) +>> +>> diff --git a/notmuch-show.c b/notmuch-show.c +>> index 195e318..4a1d699 100644 +>> --- a/notmuch-show.c +>> +++ b/notmuch-show.c +>> @@ -942,6 +942,8 @@ do_show (void *ctx, +>> +>> if (format->message_set_start) +>> fputs (format->message_set_start, stdout); +>> + if (format == &format_json) +>> + fputs ("\n", stdout); +>> +>> for (threads = notmuch_query_search_threads (query); +>> notmuch_threads_valid (threads); +>> @@ -963,6 +965,9 @@ do_show (void *ctx, +>> if (status && !res) +>> res = status; +>> +>> + if (format == &format_json) +>> + fputs ("\n", stdout); +>> + +>> notmuch_thread_destroy (thread); +>> +>> } +>> -- +>> 1.7.9.1 +>> +>> _______________________________________________ +>> notmuch mailing list +>> notmuch@notmuchmail.org +>> http://notmuchmail.org/mailman/listinfo/notmuch -- 2.26.2