Return-Path: X-Original-To: notmuch@notmuchmail.org Delivered-To: notmuch@notmuchmail.org Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id E3A326DE1B6F for ; Sun, 5 Apr 2015 16:02:46 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0.536 X-Spam-Level: X-Spam-Status: No, score=0.536 tagged_above=-999 required=5 tests=[AWL=0.526, T_HEADER_FROM_DIFFERENT_DOMAINS=0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QTD1-lcWU3Wt for ; Sun, 5 Apr 2015 16:02:44 -0700 (PDT) Received: from mx.xen14.node3324.gplhost.com (gitolite.debian.net [87.98.215.224]) by arlo.cworth.org (Postfix) with ESMTPS id 3BACB6DE1B7C for ; Sun, 5 Apr 2015 16:02:44 -0700 (PDT) Received: from remotemail by mx.xen14.node3324.gplhost.com with local (Exim 4.80) (envelope-from ) id 1YetYI-0002pZ-I1; Sun, 05 Apr 2015 23:02:02 +0000 Received: (nullmailer pid 2247 invoked by uid 1000); Sun, 05 Apr 2015 22:59:25 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [WIP2 06/12] cli/show: add extra element to structured output for metadata Date: Mon, 6 Apr 2015 07:59:08 +0900 Message-Id: <1428274754-1698-7-git-send-email-david@tethera.net> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1428274754-1698-1-git-send-email-david@tethera.net> References: <1428274754-1698-1-git-send-email-david@tethera.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Apr 2015 23:02:47 -0000 Initially this will contain the database uuid and last revision, and a query_type describing the remaining elements The changes to emacs/notmuch-tree.el and test/* are to (temporarily) force these to continue to use the old format. One non-trivial thing already tested is that the text and mbox formats do not change in format version 3. --- Makefile.local | 1 + devel/schemata | 14 ++++++++++++-- emacs/notmuch-tree.el | 2 +- lib/notmuch.h | 6 ++++++ lib/query.cc | 6 ++++++ notmuch-client.h | 4 ++-- notmuch-show.c | 4 ++-- test/T070-insert.sh | 2 +- test/T160-json.sh | 10 +++++----- test/T170-sexp.sh | 10 +++++----- test/T190-multipart.sh | 4 ++-- test/T340-maildir-sync.sh | 2 +- test/T350-crypto.sh | 14 +++++++------- test/T470-missing-headers.sh | 2 +- test/T510-thread-replies.sh | 10 +++++----- test/test-lib.sh | 4 ++++ 16 files changed, 61 insertions(+), 34 deletions(-) diff --git a/Makefile.local b/Makefile.local index 6d54742..8e17c13 100644 --- a/Makefile.local +++ b/Makefile.local @@ -290,6 +290,7 @@ notmuch_client_srcs = \ sprinter-json.c \ sprinter-sexp.c \ sprinter-text.c \ + sprinter-utils.c \ query-string.c \ mime-node.c \ crypto.c \ diff --git a/devel/schemata b/devel/schemata index 98a1b2b..76dad01 100644 --- a/devel/schemata +++ b/devel/schemata @@ -14,7 +14,7 @@ are interleaved. Keys are printed as keywords (symbols preceded by a colon), e.g. (:id "123" :time 54321 :from "foobar"). Null is printed as nil, true as t and false as nil. -This is version 2 of the structured output format. +This is version 3 of the structured output format. Version history --------------- @@ -26,6 +26,9 @@ v1 v2 - Added the thread_summary.query field. +v3 +- Initial dictionary of database metadata to notmuch show output + Common non-terminals -------------------- @@ -38,12 +41,19 @@ threadid = string # Message ID, sans "id:" messageid = string +database_metadata = { + query_type: string + uuid: string + lastmod: int +} + notmuch show schema ------------------- # A top-level set of threads (do_show) # Returned by notmuch show without a --part argument -thread_set = [thread*] + +thread_set = [database_metadata, thread*] # Top-level messages in a thread (show_messages) thread = [thread_node*] diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el index 8b6cd51..ca5f9b6 100644 --- a/emacs/notmuch-tree.el +++ b/emacs/notmuch-tree.el @@ -871,7 +871,7 @@ the same as for the function notmuch-tree." (notmuch-tag-clear-cache) (let ((proc (notmuch-start-notmuch "notmuch-tree" (current-buffer) #'notmuch-tree-process-sentinel - "show" "--body=false" "--format=sexp" + "show" "--body=false" "--format=sexp" "--format-version=2" message-arg search-args)) ;; Use a scratch buffer to accumulate partial output. ;; This buffer will be killed by the sentinel, which diff --git a/lib/notmuch.h b/lib/notmuch.h index b6be727..5c17d97 100644 --- a/lib/notmuch.h +++ b/lib/notmuch.h @@ -781,6 +781,12 @@ notmuch_sort_t notmuch_query_get_sort (notmuch_query_t *query); /** + * Return the associated notmuch database + */ +notmuch_database_t * +notmuch_query_get_database (const notmuch_query_t *query); + +/** * Add a tag that will be excluded from the query results by default. * This exclusion will be overridden if this tag appears explicitly in * the query. diff --git a/lib/query.cc b/lib/query.cc index 9cedb6a..61f85c0 100644 --- a/lib/query.cc +++ b/lib/query.cc @@ -654,3 +654,9 @@ notmuch_query_count_threads (notmuch_query_t *query) return count; } + +notmuch_database_t * +notmuch_query_get_database (const notmuch_query_t *query) +{ + return query->notmuch; +} diff --git a/notmuch-client.h b/notmuch-client.h index ab0d188..76dbc38 100644 --- a/notmuch-client.h +++ b/notmuch-client.h @@ -139,7 +139,7 @@ chomp_newline (char *str) * this. New (required) map fields can be added without increasing * this. */ -#define NOTMUCH_FORMAT_CUR 2 +#define NOTMUCH_FORMAT_CUR 3 /* The minimum supported structured output format version. Requests * for format versions below this will return an error. */ #define NOTMUCH_FORMAT_MIN 1 @@ -148,7 +148,7 @@ chomp_newline (char *str) * Must be between NOTMUCH_FORMAT_MIN and NOTMUCH_FORMAT_CUR, * inclusive. */ -#define NOTMUCH_FORMAT_MIN_ACTIVE 1 +#define NOTMUCH_FORMAT_MIN_ACTIVE 2 /* The output format version requested by the caller on the command * line. If no format version is requested, this will be set to diff --git a/notmuch-show.c b/notmuch-show.c index b80933a..4489ea5 100644 --- a/notmuch-show.c +++ b/notmuch-show.c @@ -1019,7 +1019,7 @@ do_show (void *ctx, if (! threads) return 1; - sp->begin_list (sp); + sprinter_start_output (sp, query, "threads"); for ( ; notmuch_threads_valid (threads); @@ -1041,7 +1041,7 @@ do_show (void *ctx, } - sp->end (sp); + sprinter_finish_output (sp); return res != NOTMUCH_STATUS_SUCCESS; } diff --git a/test/T070-insert.sh b/test/T070-insert.sh index 168345c..48d36af 100755 --- a/test/T070-insert.sh +++ b/test/T070-insert.sh @@ -38,7 +38,7 @@ cur_msg_filename=$(notmuch search --output=files "subject:insert-subject") test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename" test_begin_subtest "Insert message adds default tags" -output=$(notmuch show --format=json "subject:insert-subject") +output=$(NOTMUCH_SHOW --format=json "subject:insert-subject") expected='[[[{ "id": "'"${gen_msg_id}"'", "match": true, diff --git a/test/T160-json.sh b/test/T160-json.sh index c1cf649..46c1314 100755 --- a/test/T160-json.sh +++ b/test/T160-json.sh @@ -4,16 +4,16 @@ test_description="--format=json output" test_begin_subtest "Show message: json" add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[bcc]=\"test_suite+bcc@notmuchmail.org\"" "[reply-to]=\"test_suite+replyto@notmuchmail.org\"" "[body]=\"json-show-message\"" -output=$(notmuch show --format=json "json-show-message") +output=$(NOTMUCH_SHOW --format=json "json-show-message") test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite \", \"To\": \"Notmuch Test Suite \", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]" # This should be the same output as above. test_begin_subtest "Show message: json --body=true" -output=$(notmuch show --format=json --body=true "json-show-message") +output=$(NOTMUCH_SHOW --format=json --body=true "json-show-message") test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite \", \"To\": \"Notmuch Test Suite \", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]" test_begin_subtest "Show message: json --body=false" -output=$(notmuch show --format=json --body=false "json-show-message") +output=$(NOTMUCH_SHOW --format=json --body=false "json-show-message") test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite \", \"To\": \"Notmuch Test Suite \", \"Bcc\": \"test_suite+bcc@notmuchmail.org\", \"Reply-To\": \"test_suite+replyto@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}}, []]]]" test_begin_subtest "Search message: json" @@ -32,7 +32,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\", test_begin_subtest "Show message: json, utf-8" add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\"" -output=$(notmuch show --format=json "jsön-show-méssage") +output=$(NOTMUCH_SHOW --format=json "jsön-show-méssage") test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite \", \"To\": \"Notmuch Test Suite \", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]" test_begin_subtest "Show message: json, inline attachment filename" @@ -44,7 +44,7 @@ emacs_fcc_message \ "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\") (message-goto-eoh) (insert \"Message-ID: <$id>\n\")" -output=$(notmuch show --format=json "id:$id") +output=$(NOTMUCH_SHOW --format=json "id:$id") filename=$(notmuch search --output=files "id:$id") # Get length of README after base64-encoding, minus additional newline. attachment_length=$(( $(base64 $TEST_DIRECTORY/README | wc -c) - 1 )) diff --git a/test/T170-sexp.sh b/test/T170-sexp.sh index 667e319..6c7402a 100755 --- a/test/T170-sexp.sh +++ b/test/T170-sexp.sh @@ -4,16 +4,16 @@ test_description="--format=sexp output" test_begin_subtest "Show message: sexp" add_message "[subject]=\"sexp-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[bcc]=\"test_suite+bcc@notmuchmail.org\"" "[reply-to]=\"test_suite+replyto@notmuchmail.org\"" "[body]=\"sexp-show-message\"" -output=$(notmuch show --format=sexp "sexp-show-message") +output=$(NOTMUCH_SHOW --format=sexp "sexp-show-message") test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite \" :To \"Notmuch Test Suite \" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\"))) ())))" # This should be the same output as above. test_begin_subtest "Show message: sexp --body=true" -output=$(notmuch show --format=sexp --body=true "sexp-show-message") +output=$(NOTMUCH_SHOW --format=sexp --body=true "sexp-show-message") test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite \" :To \"Notmuch Test Suite \" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"sexp-show-message\n\"))) ())))" test_begin_subtest "Show message: sexp --body=false" -output=$(notmuch show --format=sexp --body=false "sexp-show-message") +output=$(NOTMUCH_SHOW --format=sexp --body=false "sexp-show-message") test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-subject\" :From \"Notmuch Test Suite \" :To \"Notmuch Test Suite \" :Bcc \"test_suite+bcc@notmuchmail.org\" :Reply-To \"test_suite+replyto@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\")) ())))" test_begin_subtest "Search message: sexp" @@ -23,7 +23,7 @@ test_expect_equal "$output" "((:thread \"0000000000000002\" :timestamp 946728000 test_begin_subtest "Show message: sexp, utf-8" add_message "[subject]=\"sexp-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\"" -output=$(notmuch show --format=sexp "jsön-show-méssage") +output=$(NOTMUCH_SHOW --format=sexp "jsön-show-méssage") test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :filename \"${gen_msg_filename}\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\" \"unread\") :headers (:Subject \"sexp-show-utf8-body-sübjéct\" :From \"Notmuch Test Suite \" :To \"Notmuch Test Suite \" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"text/plain\" :content \"jsön-show-méssage\n\"))) ())))" test_begin_subtest "Show message: sexp, inline attachment filename" @@ -35,7 +35,7 @@ emacs_fcc_message \ "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\") (message-goto-eoh) (insert \"Message-ID: <$id>\n\")" -output=$(notmuch show --format=sexp "id:$id") +output=$(NOTMUCH_SHOW --format=sexp "id:$id") filename=$(notmuch search --output=files "id:$id") # Get length of README after base64-encoding, minus additional newline. attachment_length=$(( $(base64 $TEST_DIRECTORY/README | wc -c) - 1 )) diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh index ad8d29e..f8b805f 100755 --- a/test/T190-multipart.sh +++ b/test/T190-multipart.sh @@ -744,11 +744,11 @@ cat <> EXPECTED.withhtml EOF test_begin_subtest "html parts excluded by default" -notmuch show --format=json id:htmlmessage > OUTPUT +NOTMUCH_SHOW --format=json id:htmlmessage > OUTPUT test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.nohtml)" test_begin_subtest "html parts included" -notmuch show --format=json --include-html id:htmlmessage > OUTPUT +NOTMUCH_SHOW --format=json --include-html id:htmlmessage > OUTPUT test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED.withhtml)" test_begin_subtest "indexes mime-type #1" diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh index 3186e70..50b1fb5 100755 --- a/test/T340-maildir-sync.sh +++ b/test/T340-maildir-sync.sh @@ -35,7 +35,7 @@ output=$(cd ${MAIL_DIR}/cur; ls -1 adding-replied*) test_expect_equal "$output" "adding-replied-tag:2,RS" test_begin_subtest "notmuch show works with renamed file (without notmuch new)" -output=$(notmuch show --format=json id:${gen_msg_id} | notmuch_json_show_sanitize) +output=$(NOTMUCH_SHOW --format=json id:${gen_msg_id} | notmuch_json_show_sanitize) test_expect_equal_json "$output" '[[[{"id": "XXXXX", "match": true, "excluded": false, diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh index 477b397..d9e02d0 100755 --- a/test/T350-crypto.sh +++ b/test/T350-crypto.sh @@ -35,7 +35,7 @@ test_expect_success 'emacs delivery of signed message' \ "(mml-secure-message-sign)"' test_begin_subtest "signature verification" -output=$(notmuch show --format=json --verify subject:"test signed message 001" \ +output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", @@ -69,7 +69,7 @@ test_begin_subtest "signature verification with full owner trust" # give the key full owner trust echo "${FINGERPRINT}:6:" | gpg --no-tty --import-ownertrust >>"$GNUPGHOME"/trust.log 2>&1 gpg --no-tty --check-trustdb >>"$GNUPGHOME"/trust.log 2>&1 -output=$(notmuch show --format=json --verify subject:"test signed message 001" \ +output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", @@ -103,7 +103,7 @@ test_expect_equal_json \ test_begin_subtest "signature verification with signer key unavailable" # move the gnupghome temporarily out of the way mv "${GNUPGHOME}"{,.bak} -output=$(notmuch show --format=json --verify subject:"test signed message 001" \ +output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", @@ -177,7 +177,7 @@ test_expect_equal \ "$expected" test_begin_subtest "decryption, --format=json" -output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ +output=$(NOTMUCH_SHOW --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", @@ -235,7 +235,7 @@ test_expect_equal_file OUTPUT TESTATTACHMENT test_begin_subtest "decryption failure with missing key" mv "${GNUPGHOME}"{,.bak} # The length of the encrypted attachment varies so must be normalized. -output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ +output=$(NOTMUCH_SHOW --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|' \ | sed -e 's|"content-length": 6[1234567890]*|"content-length": 652|') @@ -272,7 +272,7 @@ test_expect_success 'emacs delivery of encrypted + signed message' \ "(mml-secure-message-sign-encrypt)"' test_begin_subtest "decryption + signature verification" -output=$(notmuch show --format=json --decrypt subject:"test encrypted message 002" \ +output=$(NOTMUCH_SHOW --format=json --decrypt subject:"test encrypted message 002" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", @@ -327,7 +327,7 @@ y " \ | gpg --no-tty --quiet --command-fd 0 --armor --gen-revoke "0x${FINGERPRINT}!" 2>/dev/null \ | gpg --no-tty --quiet --import -output=$(notmuch show --format=json --verify subject:"test signed message 001" \ +output=$(NOTMUCH_SHOW --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", diff --git a/test/T470-missing-headers.sh b/test/T470-missing-headers.sh index cb38301..c3ee430 100755 --- a/test/T470-missing-headers.sh +++ b/test/T470-missing-headers.sh @@ -94,7 +94,7 @@ Body message}" test_begin_subtest "Show: json" -output=$(notmuch show --format=json '*' | notmuch_json_show_sanitize) +output=$(NOTMUCH_SHOW --format=json '*' | notmuch_json_show_sanitize) expected=$(notmuch_json_show_sanitize <