1 Return-Path: <tomi.ollila@iki.fi>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by arlo.cworth.org (Postfix) with ESMTP id 35BD86DE0217
\r
6 for <notmuch@notmuchmail.org>; Sat, 7 May 2016 10:37:18 -0700 (PDT)
\r
7 X-Virus-Scanned: Debian amavisd-new at cworth.org
\r
11 X-Spam-Status: No, score=0.6 tagged_above=-999 required=5 tests=[AWL=-0.052,
\r
12 SPF_NEUTRAL=0.652] autolearn=disabled
\r
13 Received: from arlo.cworth.org ([127.0.0.1])
\r
14 by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)
\r
15 with ESMTP id etCRJaI2VWbA for <notmuch@notmuchmail.org>;
\r
16 Sat, 7 May 2016 10:37:09 -0700 (PDT)
\r
17 Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])
\r
18 by arlo.cworth.org (Postfix) with ESMTP id 3A20E6DE00EB
\r
19 for <notmuch@notmuchmail.org>; Sat, 7 May 2016 10:37:09 -0700 (PDT)
\r
20 Received: from guru.guru-group.fi (localhost [IPv6:::1])
\r
21 by guru.guru-group.fi (Postfix) with ESMTP id AA6B9100063;
\r
22 Sat, 7 May 2016 20:37:07 +0300 (EEST)
\r
23 From: Tomi Ollila <tomi.ollila@iki.fi>
\r
24 To: David Bremner <david@tethera.net>, notmuch@notmuchmail.org
\r
25 Subject: Re: [Patch v3 07/11] CLI: add optional config data to dump output.
\r
26 In-Reply-To: <1462065879-29860-8-git-send-email-david@tethera.net>
\r
27 References: <1462065879-29860-1-git-send-email-david@tethera.net>
\r
28 <1462065879-29860-8-git-send-email-david@tethera.net>
\r
29 User-Agent: Notmuch/0.22+9~gb26b5ad (http://notmuchmail.org) Emacs/24.3.1
\r
30 (x86_64-unknown-linux-gnu)
\r
31 X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL
\r
32 $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F
\r
33 !)g;OY^,BjTbr)Np:%c_o'jj,Z
\r
34 Date: Sat, 07 May 2016 20:37:07 +0300
\r
35 Message-ID: <m2wpn5epmk.fsf@guru.guru-group.fi>
\r
37 Content-Type: text/plain
\r
38 X-BeenThere: notmuch@notmuchmail.org
\r
39 X-Mailman-Version: 2.1.20
\r
41 List-Id: "Use and development of the notmuch mail system."
\r
42 <notmuch.notmuchmail.org>
\r
43 List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,
\r
44 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
45 List-Archive: <http://notmuchmail.org/pipermail/notmuch/>
\r
46 List-Post: <mailto:notmuch@notmuchmail.org>
\r
47 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
48 List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,
\r
49 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
50 X-List-Received-Date: Sat, 07 May 2016 17:37:18 -0000
\r
52 On Sun, May 01 2016, David Bremner <david@tethera.net> wrote:
\r
54 > Note that it changes the default dump output format, but doesn't break
\r
55 > existing notmuch-restore. It might break user scripts though.
\r
57 > doc/man1/notmuch-dump.rst | 17 ++++++++++
\r
58 > notmuch-client.h | 8 +++++
\r
59 > notmuch-dump.c | 80 +++++++++++++++++++++++++++++++++++++++++++++--
\r
60 > notmuch-new.c | 2 +-
\r
61 > test/T150-tagging.sh | 8 ++---
\r
62 > test/T240-dump-restore.sh | 14 ++++-----
\r
63 > test/T590-libconfig.sh | 17 ++++++++++
\r
64 > test/test-lib.sh | 6 ++++
\r
65 > 8 files changed, 137 insertions(+), 15 deletions(-)
\r
67 > diff --git a/doc/man1/notmuch-dump.rst b/doc/man1/notmuch-dump.rst
\r
68 > index a37c337..eda9e07 100644
\r
69 > --- a/doc/man1/notmuch-dump.rst
\r
70 > +++ b/doc/man1/notmuch-dump.rst
\r
71 > @@ -71,6 +71,23 @@ Supported options for **dump** include
\r
72 > characters. Note also that tags with spaces will not be
\r
73 > correctly restored with this format.
\r
75 > + ``--include=(config|tags)``
\r
77 > + Control what kind of metadata is included in the output.
\r
81 > + Output configuration data stored in the database. Each line
\r
82 > + starts with "#@ ", followed by a space seperated key-value
\r
83 > + pair. Both key and value are hex encoded if needed.
\r
87 > + Output per-message metadata, namely tags. See *format* above
\r
88 > + for description of the output.
\r
90 > + The default is to include both tags and configuration information
\r
92 > ``--output=``\ <filename>
\r
93 > Write output to given file instead of stdout.
\r
95 > diff --git a/notmuch-client.h b/notmuch-client.h
\r
96 > index b3d0b66..ae6f124 100644
\r
97 > --- a/notmuch-client.h
\r
98 > +++ b/notmuch-client.h
\r
99 > @@ -446,11 +446,19 @@ typedef enum dump_formats {
\r
103 > +typedef enum dump_includes {
\r
104 > + DUMP_INCLUDE_TAGS=1,
\r
105 > + DUMP_INCLUDE_CONFIG=2,
\r
106 > +} dump_include_t;
\r
108 > +#define NOTMUCH_DUMP_VERSION 2
\r
111 > notmuch_database_dump (notmuch_database_t *notmuch,
\r
112 > const char *output_file_name,
\r
113 > const char *query_str,
\r
114 > dump_format_t output_format,
\r
115 > + dump_include_t include,
\r
116 > notmuch_bool_t gzip_output);
\r
118 > /* If status is non-zero (i.e. error) print appropriate
\r
119 > diff --git a/notmuch-dump.c b/notmuch-dump.c
\r
120 > index 829781f..a6cf810 100644
\r
121 > --- a/notmuch-dump.c
\r
122 > +++ b/notmuch-dump.c
\r
123 > @@ -23,16 +23,82 @@
\r
124 > #include "string-util.h"
\r
125 > #include <zlib.h>
\r
128 > +database_dump_config (notmuch_database_t *notmuch, gzFile output)
\r
130 > + notmuch_config_list_t *list;
\r
131 > + int ret = EXIT_FAILURE;
\r
132 > + char *buffer = NULL;
\r
133 > + size_t buffer_size = 0;
\r
135 > + if (print_status_database ("notmuch dump", notmuch,
\r
136 > + notmuch_database_get_config_list (notmuch, NULL, &list)))
\r
139 > + for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
\r
140 > + if (hex_encode (notmuch, notmuch_config_list_key (list),
\r
141 > + &buffer, &buffer_size) != HEX_SUCCESS) {
\r
142 > + fprintf (stderr, "Error: failed to hex-encode config key %s\n",
\r
143 > + notmuch_config_list_key (list));
\r
146 > + gzprintf (output, "#@ %s", buffer);
\r
148 > + if (hex_encode (notmuch, notmuch_config_list_value (list),
\r
149 > + &buffer, &buffer_size) != HEX_SUCCESS) {
\r
150 > + fprintf (stderr, "Error: failed to hex-encode config value %s\n",
\r
151 > + notmuch_config_list_value (list) );
\r
155 > + gzprintf (output, " %s\n", buffer);
\r
158 > + ret = EXIT_SUCCESS;
\r
162 > + notmuch_config_list_destroy (list);
\r
165 > + talloc_free (buffer);
\r
171 > +print_dump_header (gzFile output, int output_format, int include)
\r
173 > + gzprintf (output, "#notmuch-dump %s:%d %s%s%s\n",
\r
174 > + (output_format == DUMP_FORMAT_SUP) ? "sup" : "batch-tag",
\r
175 > + NOTMUCH_DUMP_VERSION,
\r
176 > + (include & DUMP_INCLUDE_CONFIG) ? "config" : "",
\r
177 > + (include & DUMP_INCLUDE_TAGS) && (include & DUMP_INCLUDE_CONFIG) ? "," : "",
\r
178 > + (include & DUMP_INCLUDE_TAGS) ? "tags" : "");
\r
184 > database_dump_file (notmuch_database_t *notmuch, gzFile output,
\r
185 > - const char *query_str, int output_format)
\r
186 > + const char *query_str, int output_format, int include)
\r
188 > notmuch_query_t *query;
\r
189 > notmuch_messages_t *messages;
\r
190 > notmuch_message_t *message;
\r
191 > notmuch_tags_t *tags;
\r
193 > + print_dump_header (output, output_format, include);
\r
195 > + if (include & DUMP_INCLUDE_CONFIG) {
\r
196 > + if (print_status_database ("notmuch dump", notmuch,
\r
197 > + database_dump_config(notmuch,output)))
\r
198 > + return EXIT_FAILURE;
\r
201 > + if (! (include & DUMP_INCLUDE_TAGS))
\r
202 > + return EXIT_SUCCESS;
\r
207 > @@ -130,6 +196,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
\r
208 > const char *output_file_name,
\r
209 > const char *query_str,
\r
210 > dump_format_t output_format,
\r
211 > + dump_include_t include,
\r
212 > notmuch_bool_t gzip_output)
\r
214 > gzFile output = NULL;
\r
215 > @@ -164,7 +231,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
\r
219 > - ret = database_dump_file (notmuch, output, query_str, output_format);
\r
220 > + ret = database_dump_file (notmuch, output, query_str, output_format, include);
\r
221 > if (ret) goto DONE;
\r
223 > ret = gzflush (output, Z_FINISH);
\r
224 > @@ -226,6 +293,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
\r
227 > int output_format = DUMP_FORMAT_BATCH_TAG;
\r
228 > + int include = 0;
\r
229 > notmuch_bool_t gzip_output = 0;
\r
231 > notmuch_opt_desc_t options[] = {
\r
232 > @@ -233,6 +301,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
\r
233 > (notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
\r
234 > { "batch-tag", DUMP_FORMAT_BATCH_TAG },
\r
236 > + { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
\r
237 > + (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
\r
238 > + { "tags", DUMP_INCLUDE_TAGS} } },
\r
239 > { NOTMUCH_OPT_STRING, &output_file_name, "output", 'o', 0 },
\r
240 > { NOTMUCH_OPT_BOOLEAN, &gzip_output, "gzip", 'z', 0 },
\r
241 > { NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
\r
242 > @@ -245,6 +316,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
\r
244 > notmuch_process_shared_options (argv[0]);
\r
246 > + if (include == 0)
\r
247 > + include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
\r
249 > if (opt_index < argc) {
\r
250 > query_str = query_string_from_args (notmuch, argc - opt_index, argv + opt_index);
\r
251 > if (query_str == NULL) {
\r
252 > @@ -254,7 +328,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
\r
255 > ret = notmuch_database_dump (notmuch, output_file_name, query_str,
\r
256 > - output_format, gzip_output);
\r
257 > + output_format, include, gzip_output);
\r
259 > notmuch_database_destroy (notmuch);
\r
261 > diff --git a/notmuch-new.c b/notmuch-new.c
\r
262 > index 04cb5ca..196a36e 100644
\r
263 > --- a/notmuch-new.c
\r
264 > +++ b/notmuch-new.c
\r
265 > @@ -1045,7 +1045,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
\r
268 > if (notmuch_database_dump (notmuch, backup_name, "",
\r
269 > - DUMP_FORMAT_BATCH_TAG, TRUE)) {
\r
270 > + DUMP_FORMAT_BATCH_TAG, DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS, TRUE)) {
\r
271 > fprintf (stderr, "Backup failed. Aborting upgrade.");
\r
272 > return EXIT_FAILURE;
\r
274 > diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
\r
275 > index a451ffa..61d1311 100755
\r
276 > --- a/test/T150-tagging.sh
\r
277 > +++ b/test/T150-tagging.sh
\r
278 > @@ -188,7 +188,7 @@ cat <<EOF > EXPECTED
\r
279 > +%22%27%22%27%22%22%27%27 +inbox +tag4 +tag5 +unread -- id:msg-002@notmuch-test-suite
\r
282 > -notmuch dump --format=batch-tag | sort > OUTPUT
\r
283 > +NOTMUCH_DUMP_TAGS > OUTPUT
\r
284 > notmuch restore --format=batch-tag < BACKUP
\r
285 > test_expect_equal_file EXPECTED OUTPUT
\r
287 > @@ -209,7 +209,7 @@ cat <<EOF > EXPECTED
\r
288 > +%21@%23%20%24%25%5e%26%2a%29-_=+%5b%7b%5c%20%7c%3b%3a%27%20%22,.%3c%60%7e +inbox +tag5 +unread -- id:msg-001@notmuch-test-suite
\r
291 > -notmuch dump --format=batch-tag | sort > OUTPUT
\r
292 > +NOTMUCH_DUMP_TAGS > OUTPUT
\r
293 > notmuch restore --format=batch-tag < BACKUP
\r
294 > test_expect_equal_file EXPECTED OUTPUT
\r
296 > @@ -235,7 +235,7 @@ cat <<EOF > EXPECTED
\r
297 > +%2a@%7d%cf%b5%f4%85%80%adO3%da%a7 +=%e0%ac%95%c8%b3+%ef%aa%95%c8%a64w%c7%9d%c9%a2%cf%b3%d6%82%24B%c4%a9%c5%a1UX%ee%99%b0%27E7%ca%a4%d0%8b%5d +A%e1%a0%bc%de%8b%d5%b2V%d9%9b%f3%b5%a2%a3M%d8%a1u@%f0%a0%ac%948%7e%f0%ab%86%af%27 +L%df%85%ef%a1%a5m@%d3%96%c2%ab%d4%9f%ca%b8%f3%b3%a2%bf%c7%b1_u%d7%b4%c7%b1 +P%c4%98%2f +R +inbox +tag5 +unread +%7e%d1%8b%25%ec%a0%ae%d1%a0M%3b%e3%b6%b7%e9%a4%87%3c%db%9a%cc%a8%e1%96%9d +%c4%bf7%c7%ab9H%c4%99k%ea%91%bd%c3%8ck%e2%b3%8dk%c5%952V%e4%99%b2%d9%b3%e4%8b%bda%5b%24%c7%9b +%da%88=f%cc%b9I%ce%af%7b%c9%97%e3%b9%8bH%cb%92X%d2%8c6 +%dc%9crh%d2%86B%e5%97%a2%22t%ed%99%82d -- id:msg-001@notmuch-test-suite
\r
300 > -notmuch dump --format=batch-tag | sort > OUTPUT
\r
301 > +NOTMUCH_DUMP_TAGS > OUTPUT
\r
302 > notmuch restore --format=batch-tag < BACKUP
\r
303 > test_expect_equal_file EXPECTED OUTPUT
\r
305 > @@ -260,7 +260,7 @@ cat <<EOF > EXPECTED
\r
306 > +foo%3a%3abar%25 +found%3a%3ait +inbox +tag5 +unread +winner -- id:msg-001@notmuch-test-suite
\r
309 > -notmuch dump --format=batch-tag | sort > OUTPUT
\r
310 > +NOTMUCH_DUMP_TAGS > OUTPUT
\r
311 > notmuch restore --format=batch-tag < BACKUP
\r
312 > test_expect_equal_file EXPECTED OUTPUT
\r
314 > diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
\r
315 > index e6976ff..758d355 100755
\r
316 > --- a/test/T240-dump-restore.sh
\r
317 > +++ b/test/T240-dump-restore.sh
\r
318 > @@ -97,7 +97,7 @@ test_expect_equal_file dump.expected dump.actual
\r
319 > # Note, we assume all messages from cworth have a message-id
\r
320 > # containing cworth.org
\r
322 > -grep 'cworth[.]org' dump.expected > dump-cworth.expected
\r
323 > +(head -1 dump.expected ; grep 'cworth[.]org' dump.expected) > dump-cworth.expected
\r
325 Instead of executing the commands in subshell, the 'group command' format
\r
326 could be used (and run in current shell). i.e.
\r
328 { head -1 dump.expected ; grep 'cworth[.]org' dump.expected; } > dump-cworth.expected
\r
330 > test_begin_subtest "dump -- from:cworth"
\r
331 > notmuch dump -- from:cworth > dump-dash-cworth.actual
\r
332 > @@ -118,16 +118,16 @@ notmuch search --output=messages from:cworth | sed s/^id:// |\
\r
333 > test_expect_equal_file OUTPUT EXPECTED
\r
335 > test_begin_subtest "format=batch-tag, dump sanity check."
\r
336 > -notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
\r
337 > +NOTMUCH_DUMP_TAGS --format=sup from:cworth | cut -f1 -d' ' | \
\r
338 > sort > EXPECTED.$test_count
\r
339 > -notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
\r
340 > +NOTMUCH_DUMP_TAGS --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
\r
341 > sort > OUTPUT.$test_count
\r
342 > test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
\r
344 > test_begin_subtest "format=batch-tag, missing newline"
\r
345 > printf "+a_tag_without_newline -- id:20091117232137.GA7669@griffis1.net" > IN
\r
346 > notmuch restore --accumulate < IN
\r
347 > -notmuch dump id:20091117232137.GA7669@griffis1.net > OUT
\r
348 > +NOTMUCH_DUMP_TAGS id:20091117232137.GA7669@griffis1.net > OUT
\r
349 > cat <<EOF > EXPECTED
\r
350 > +a_tag_without_newline +inbox +unread -- id:20091117232137.GA7669@griffis1.net
\r
352 > @@ -155,7 +155,7 @@ cat <<EOF >EXPECTED.$test_count
\r
353 > + -- id:20091117232137.GA7669@griffis1.net
\r
355 > notmuch restore --format=batch-tag < EXPECTED.$test_count
\r
356 > -notmuch dump --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
\r
357 > +NOTMUCH_DUMP_TAGS --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
\r
358 > test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
\r
360 > tag1='comic_swear=$&^%$^%\\//-+$^%$'
\r
361 > @@ -217,9 +217,9 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
\r
362 > test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
\r
364 > test_begin_subtest 'format=batch-tag, checking encoded output'
\r
365 > -notmuch dump --format=batch-tag -- from:cworth |\
\r
366 > +NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
\r
367 > awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
\r
369 The above I don't understand, $enc1, $enc2 and $enc3 should be expanded by
\r
370 the shell into the awk command -- I don't see these set anywhere, so to
\r
371 me it looks like these are empty strings...
\r
373 > -notmuch dump --format=batch-tag -- from:cworth > OUTPUT.$test_count
\r
374 > +NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
\r
375 > test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
\r
377 > test_begin_subtest 'restoring sane tags'
\r
378 > diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
\r
379 > index 8ca6883..5ea5300 100755
\r
380 > --- a/test/T590-libconfig.sh
\r
381 > +++ b/test/T590-libconfig.sh
\r
382 > @@ -115,4 +115,21 @@ testkey2 testvalue2
\r
384 > test_expect_equal_file EXPECTED OUTPUT
\r
386 > +test_begin_subtest "dump config"
\r
387 > +cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
\r
389 > + RUN(notmuch_database_set_config (db, "key with spaces", "value, with, spaces!"));
\r
392 > +notmuch dump --include=config >OUTPUT
\r
393 > +cat <<'EOF' >EXPECTED
\r
394 > +#notmuch-dump batch-tag:2 config
\r
395 > +#@ aaabefore beforeval
\r
396 > +#@ key%20with%20spaces value,%20with,%20spaces%21
\r
397 > +#@ testkey1 testvalue1
\r
398 > +#@ testkey2 testvalue2
\r
399 > +#@ zzzafter afterval
\r
401 > +test_expect_equal_file EXPECTED OUTPUT
\r
404 > diff --git a/test/test-lib.sh b/test/test-lib.sh
\r
405 > index 09f8731..68a8774 100644
\r
406 > --- a/test/test-lib.sh
\r
407 > +++ b/test/test-lib.sh
\r
408 > @@ -673,6 +673,12 @@ NOTMUCH_NEW ()
\r
409 > notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
\r
412 > +NOTMUCH_DUMP_TAGS ()
\r
414 > + # this relies on the default format being batch-tag, otherwise some tests will break
\r
415 > + notmuch dump --include=tags "${@}" | sed '/^#/d' | sort
\r
418 > notmuch_search_sanitize ()
\r
420 > perl -pe 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
\r
424 > _______________________________________________
\r
425 > notmuch mailing list
\r
426 > notmuch@notmuchmail.org
\r
427 > https://notmuchmail.org/mailman/listinfo/notmuch
\r