--- /dev/null
+Return-Path: <dkg@fifthhorseman.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 21D6D6DE01BA\r
+ for <notmuch@notmuchmail.org>; Fri, 8 Jul 2016 03:13:17 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 4gR5kbHFHJCr for <notmuch@notmuchmail.org>;\r
+ Fri, 8 Jul 2016 03:13:08 -0700 (PDT)\r
+Received: from che.mayfirst.org (che.mayfirst.org [162.247.75.118])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 0ED3E6DE00BD\r
+ for <notmuch@notmuchmail.org>; Fri, 8 Jul 2016 03:13:07 -0700 (PDT)\r
+Received: from fifthhorseman.net (unknown [88.128.80.54])\r
+ by che.mayfirst.org (Postfix) with ESMTPSA id 502AEF98B\r
+ for <notmuch@notmuchmail.org>; Fri, 8 Jul 2016 06:13:03 -0400 (EDT)\r
+Received: by fifthhorseman.net (Postfix, from userid 1000)\r
+ id CE17F21753; Fri, 8 Jul 2016 11:27:34 +0200 (CEST)\r
+From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>\r
+To: Notmuch Mail <notmuch@notmuchmail.org>\r
+Subject: [PATCH v4 16/16] add "notmuch reindex" subcommand\r
+Date: Fri, 8 Jul 2016 11:27:27 +0200\r
+Message-Id: <1467970047-8013-17-git-send-email-dkg@fifthhorseman.net>\r
+X-Mailer: git-send-email 2.8.1\r
+In-Reply-To: <1467970047-8013-1-git-send-email-dkg@fifthhorseman.net>\r
+References: <1467970047-8013-1-git-send-email-dkg@fifthhorseman.net>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=UTF-8\r
+Content-Transfer-Encoding: 8bit\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 08 Jul 2016 10:13:17 -0000\r
+\r
+This new subcommand takes a set of search terms, and re-indexes the\r
+list of matching messages using the supplied options.\r
+\r
+This can be used to index the cleartext of encrypted messages with\r
+something like:\r
+\r
+ notmuch reindex --try-decrypt \\r
+ tag:encrypted and not has:index-decryption=success\r
+---\r
+ Makefile.local | 1 +\r
+ doc/conf.py | 7 ++\r
+ doc/index.rst | 1 +\r
+ doc/man1/notmuch-reindex.rst | 41 ++++++++++\r
+ doc/man1/notmuch.rst | 1 +\r
+ doc/man7/notmuch-search-terms.rst | 7 +-\r
+ notmuch-client.h | 3 +\r
+ notmuch-reindex.c | 152 ++++++++++++++++++++++++++++++++++++++\r
+ notmuch.c | 2 +\r
+ test/T357-index-decryption.sh | 67 +++++++++++++++++\r
+ 10 files changed, 280 insertions(+), 2 deletions(-)\r
+ create mode 100644 doc/man1/notmuch-reindex.rst\r
+ create mode 100644 notmuch-reindex.c\r
+\r
+diff --git a/Makefile.local b/Makefile.local\r
+index c13ba76..a916bd0 100644\r
+--- a/Makefile.local\r
++++ b/Makefile.local\r
+@@ -281,6 +281,7 @@ notmuch_client_srcs = \\r
+ notmuch-dump.c \\r
+ notmuch-insert.c \\r
+ notmuch-new.c \\r
++ notmuch-reindex.c \\r
+ notmuch-reply.c \\r
+ notmuch-restore.c \\r
+ notmuch-search.c \\r
+diff --git a/doc/conf.py b/doc/conf.py\r
+index 8b93296..8455a51 100644\r
+--- a/doc/conf.py\r
++++ b/doc/conf.py\r
+@@ -94,6 +94,10 @@ man_pages = [\r
+ u'incorporate new mail into the notmuch database',\r
+ [u'Carl Worth and many others'], 1),\r
+ \r
++('man1/notmuch-reindex','notmuch-reindex',\r
++ u're-index matching messages',\r
++ [u'Carl Worth and many others'], 1),\r
++\r
+ ('man1/notmuch-reply','notmuch-reply',\r
+ u'constructs a reply template for a set of messages',\r
+ [u'Carl Worth and many others'], 1),\r
+@@ -162,6 +166,9 @@ texinfo_documents = [\r
+ ('man1/notmuch-new','notmuch-new',u'notmuch Documentation',\r
+ u'Carl Worth and many others', 'notmuch-new',\r
+ 'incorporate new mail into the notmuch database','Miscellaneous'),\r
++('man1/notmuch-reindex','notmuch-reindex',u'notmuch Documentation',\r
++ u'Carl Worth and many others', 'notmuch-reindex',\r
++ 're-index matching messages','Miscellaneous'),\r
+ ('man1/notmuch-reply','notmuch-reply',u'notmuch Documentation',\r
+ u'Carl Worth and many others', 'notmuch-reply',\r
+ 'constructs a reply template for a set of messages','Miscellaneous'),\r
+diff --git a/doc/index.rst b/doc/index.rst\r
+index 344606d..aa6c9f4 100644\r
+--- a/doc/index.rst\r
++++ b/doc/index.rst\r
+@@ -18,6 +18,7 @@ Contents:\r
+ man5/notmuch-hooks\r
+ man1/notmuch-insert\r
+ man1/notmuch-new\r
++ man1/notmuch-reindex\r
+ man1/notmuch-reply\r
+ man1/notmuch-restore\r
+ man1/notmuch-search\r
+diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst\r
+new file mode 100644\r
+index 0000000..7ccc947\r
+--- /dev/null\r
++++ b/doc/man1/notmuch-reindex.rst\r
+@@ -0,0 +1,41 @@\r
++===========\r
++notmuch-reindex\r
++===========\r
++\r
++SYNOPSIS\r
++========\r
++\r
++**notmuch** **reindex** [*option* ...] <*search-term*> ...\r
++\r
++DESCRIPTION\r
++===========\r
++\r
++Re-index all messages matching the search terms.\r
++\r
++See **notmuch-search-terms(7)** for details of the supported syntax for\r
++<*search-term*\ >.\r
++\r
++The **reindex** command searches for all messages matching the\r
++supplied search terms, and re-creates the full-text index on these\r
++messages using the supplied options.\r
++\r
++Supported options for **reindex** include\r
++\r
++ ``--try-decrypt``\r
++\r
++ For each message, if it is encrypted, try to decrypt it while\r
++ indexing. If decryption is successful, index the cleartext\r
++ itself. Be aware that the index is likely sufficient to\r
++ reconstruct the cleartext of the message itself, so please\r
++ ensure that the notmuch message index is adequately\r
++ protected. DO NOT USE THIS FLAG without considering the\r
++ security of your index.\r
++\r
++SEE ALSO\r
++========\r
++\r
++**notmuch(1)**, **notmuch-config(1)**, **notmuch-count(1)**,\r
++**notmuch-dump(1)**, **notmuch-hooks(5)**, **notmuch-insert(1)**,\r
++**notmuch-new(1)**,\r
++**notmuch-reply(1)**, **notmuch-restore(1)**, **notmuch-search(1)**,\r
++**notmuch-search-terms(7)**, **notmuch-show(1)**, **notmuch-tag(1)**\r
+diff --git a/doc/man1/notmuch.rst b/doc/man1/notmuch.rst\r
+index edd04ef..e6a14f6 100644\r
+--- a/doc/man1/notmuch.rst\r
++++ b/doc/man1/notmuch.rst\r
+@@ -140,6 +140,7 @@ SEE ALSO\r
+ \r
+ **notmuch-config(1)**, **notmuch-count(1)**, **notmuch-dump(1)**,\r
+ **notmuch-hooks(5)**, **notmuch-insert(1)**, **notmuch-new(1)**,\r
++**notmuch-reindex(1)**,\r
+ **notmuch-reply(1)**, **notmuch-restore(1)**, **notmuch-search(1)**,\r
+ **notmuch-search-terms(7)**, **notmuch-show(1)**, **notmuch-tag(1)**,\r
+ **notmuch-address(1)**\r
+diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst\r
+index 075f88c..e5ea899 100644\r
+--- a/doc/man7/notmuch-search-terms.rst\r
++++ b/doc/man7/notmuch-search-terms.rst\r
+@@ -9,6 +9,8 @@ SYNOPSIS\r
+ \r
+ **notmuch** **dump** [--format=(batch-tag|sup)] [--] [--output=<*file*>] [--] [<*search-term*> ...]\r
+ \r
++**notmuch** **reindex** [option ...] <*search-term*> ...\r
++\r
+ **notmuch** **search** [option ...] <*search-term*> ...\r
+ \r
+ **notmuch** **show** [option ...] <*search-term*> ...\r
+@@ -395,5 +397,6 @@ SEE ALSO\r
+ \r
+ **notmuch(1)**, **notmuch-config(1)**, **notmuch-count(1)**,\r
+ **notmuch-dump(1)**, **notmuch-hooks(5)**, **notmuch-insert(1)**,\r
+-**notmuch-new(1)**, **notmuch-reply(1)**, **notmuch-restore(1)**,\r
+-**notmuch-search(1)**, **notmuch-show(1)**, **notmuch-tag(1)**\r
++**notmuch-new(1)**, **notmuch-reindex(1)**, **notmuch-reply(1)**,\r
++**notmuch-restore(1)**, **notmuch-search(1)**, **notmuch-show(1)**,\r
++**notmuch-tag(1)**\r
+diff --git a/notmuch-client.h b/notmuch-client.h\r
+index a65bb6d..a32ef7a 100644\r
+--- a/notmuch-client.h\r
++++ b/notmuch-client.h\r
+@@ -169,6 +169,9 @@ int\r
+ notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[]);\r
+ \r
+ int\r
++notmuch_reindex_command (notmuch_config_t *config, int argc, char *argv[]);\r
++\r
++int\r
+ notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[]);\r
+ \r
+ int\r
+diff --git a/notmuch-reindex.c b/notmuch-reindex.c\r
+new file mode 100644\r
+index 0000000..6fc88c5\r
+--- /dev/null\r
++++ b/notmuch-reindex.c\r
+@@ -0,0 +1,152 @@\r
++/* notmuch - Not much of an email program, (just index and search)\r
++ *\r
++ * Copyright © 2016 Daniel Kahn Gillmor\r
++ *\r
++ * This program is free software: you can redistribute it and/or modify\r
++ * it under the terms of the GNU General Public License as published by\r
++ * the Free Software Foundation, either version 3 of the License, or\r
++ * (at your option) any later version.\r
++ *\r
++ * This program is distributed in the hope that it will be useful,\r
++ * but WITHOUT ANY WARRANTY; without even the implied warranty of\r
++ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r
++ * GNU General Public License for more details.\r
++ *\r
++ * You should have received a copy of the GNU General Public License\r
++ * along with this program. If not, see http://www.gnu.org/licenses/ .\r
++ *\r
++ * Author: Daniel Kahn Gillmor <dkg@fifthhorseman.net>\r
++ */\r
++\r
++#include "notmuch-client.h"\r
++#include "string-util.h"\r
++\r
++static volatile sig_atomic_t interrupted;\r
++\r
++static void\r
++handle_sigint (unused (int sig))\r
++{\r
++ static char msg[] = "Stopping... \n";\r
++\r
++ /* This write is "opportunistic", so it's okay to ignore the\r
++ * result. It is not required for correctness, and if it does\r
++ * fail or produce a short write, we want to get out of the signal\r
++ * handler as quickly as possible, not retry it. */\r
++ IGNORE_RESULT (write (2, msg, sizeof (msg) - 1));\r
++ interrupted = 1;\r
++}\r
++\r
++/* reindex all messages matching 'query_string' using the passed-in indexopts\r
++ */\r
++static int\r
++reindex_query (notmuch_database_t *notmuch, const char *query_string,\r
++ notmuch_indexopts_t *indexopts)\r
++{\r
++ notmuch_query_t *query;\r
++ notmuch_messages_t *messages;\r
++ notmuch_message_t *message;\r
++ notmuch_status_t status;\r
++\r
++ int ret = NOTMUCH_STATUS_SUCCESS;\r
++\r
++ query = notmuch_query_create (notmuch, query_string);\r
++ if (query == NULL) {\r
++ fprintf (stderr, "Out of memory.\n");\r
++ return 1;\r
++ }\r
++\r
++ /* reindexing is not interested in any special sort order */\r
++ notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);\r
++\r
++ status = notmuch_query_search_messages_st (query, &messages);\r
++ if (print_status_query ("notmuch reindex", query, status))\r
++ return status;\r
++\r
++ for (;\r
++ notmuch_messages_valid (messages) && ! interrupted;\r
++ notmuch_messages_move_to_next (messages)) {\r
++ message = notmuch_messages_get (messages);\r
++\r
++ notmuch_message_reindex(message, indexopts);\r
++ notmuch_message_destroy (message);\r
++ if (ret != NOTMUCH_STATUS_SUCCESS)\r
++ break;\r
++ }\r
++\r
++ notmuch_query_destroy (query);\r
++\r
++ return ret || interrupted;\r
++}\r
++\r
++int\r
++notmuch_reindex_command (notmuch_config_t *config, int argc, char *argv[])\r
++{\r
++ char *query_string = NULL;\r
++ notmuch_database_t *notmuch;\r
++ struct sigaction action;\r
++ notmuch_bool_t try_decrypt = FALSE;\r
++ int opt_index;\r
++ int ret;\r
++ notmuch_status_t status;\r
++ notmuch_indexopts_t *indexopts = NULL;\r
++\r
++ /* Set up our handler for SIGINT */\r
++ memset (&action, 0, sizeof (struct sigaction));\r
++ action.sa_handler = handle_sigint;\r
++ sigemptyset (&action.sa_mask);\r
++ action.sa_flags = SA_RESTART;\r
++ sigaction (SIGINT, &action, NULL);\r
++\r
++ notmuch_opt_desc_t options[] = {\r
++ { NOTMUCH_OPT_BOOLEAN, &try_decrypt, "try-decrypt", 0, 0 },\r
++ { NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },\r
++ { 0, 0, 0, 0, 0 }\r
++ };\r
++\r
++ opt_index = parse_arguments (argc, argv, options, 1);\r
++ if (opt_index < 0)\r
++ return EXIT_FAILURE;\r
++\r
++ notmuch_process_shared_options (argv[0]);\r
++\r
++ if (notmuch_database_open (notmuch_config_get_database_path (config),\r
++ NOTMUCH_DATABASE_MODE_READ_WRITE, ¬much))\r
++ return EXIT_FAILURE;\r
++\r
++ notmuch_exit_if_unmatched_db_uuid (notmuch);\r
++\r
++ indexopts = notmuch_indexopts_create();\r
++ if (!indexopts)\r
++ return EXIT_FAILURE;\r
++\r
++ status = notmuch_indexopts_set_try_decrypt (indexopts, try_decrypt);\r
++ if (status)\r
++ fprintf (stderr, "Warning: failed to set --try-decrypt to %d (%s)\n",\r
++ try_decrypt, notmuch_status_to_string (status));\r
++\r
++ if (try_decrypt) {\r
++ const char* gpg_path = notmuch_config_get_crypto_gpg_path (config);\r
++ status = notmuch_indexopts_set_gpg_path (indexopts, gpg_path);\r
++ if (status)\r
++ fprintf (stderr, "Warning: failed to set gpg_path for reindexing to '%s' (%s)\n",\r
++ gpg_path ? gpg_path : "(NULL)",\r
++ notmuch_status_to_string (status));\r
++ }\r
++\r
++ query_string = query_string_from_args (config, argc-opt_index, argv+opt_index);\r
++ if (query_string == NULL) {\r
++ fprintf (stderr, "Out of memory\n");\r
++ return EXIT_FAILURE;\r
++ }\r
++\r
++ if (*query_string == '\0') {\r
++ fprintf (stderr, "Error: notmuch reindex requires at least one search term.\n");\r
++ return EXIT_FAILURE;\r
++ }\r
++ \r
++ ret = reindex_query (notmuch, query_string, indexopts);\r
++\r
++ notmuch_database_destroy (notmuch);\r
++\r
++ return ret || interrupted ? EXIT_FAILURE : EXIT_SUCCESS;\r
++}\r
+diff --git a/notmuch.c b/notmuch.c\r
+index 38b73c1..5aadea5 100644\r
+--- a/notmuch.c\r
++++ b/notmuch.c\r
+@@ -123,6 +123,8 @@ static command_t commands[] = {\r
+ "Restore the tags from the given dump file (see 'dump')." },\r
+ { "compact", notmuch_compact_command, FALSE,\r
+ "Compact the notmuch database." },\r
++ { "reindex", notmuch_reindex_command, FALSE,\r
++ "Re-index all messages matching the search terms." },\r
+ { "config", notmuch_config_command, FALSE,\r
+ "Get or set settings in the notmuch configuration file." },\r
+ { "help", notmuch_help_command, TRUE, /* create but don't save config */\r
+diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh\r
+index 27d6885..9510c57 100755\r
+--- a/test/T357-index-decryption.sh\r
++++ b/test/T357-index-decryption.sh\r
+@@ -46,4 +46,71 @@ test_expect_equal \\r
+ "$output" \\r
+ "$expected"\r
+ \r
++# add a tag to all messages to ensure that it stays after reindexing\r
++test_expect_success 'tagging all messages' \\r
++ 'notmuch tag +blarney "encrypted message"'\r
++test_begin_subtest "verify that tags have not changed"\r
++output=$(notmuch search tag:blarney)\r
++expected='thread:0000000000000001 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox)\r
++thread:0000000000000002 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox)'\r
++test_expect_equal \\r
++ "$output" \\r
++ "$expected"\r
++\r
++# see if first message shows up after reindexing with --try-decrypt\r
++test_expect_success 'reindex old messages' \\r
++ 'notmuch reindex --try-decrypt tag:encrypted and not has:index-decryption=success'\r
++test_begin_subtest "reindexed encrypted message, including cleartext"\r
++output=$(notmuch search wumpus)\r
++expected='thread:0000000000000002 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox)\r
++thread:0000000000000003 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox)'\r
++test_expect_equal \\r
++ "$output" \\r
++ "$expected"\r
++\r
++# and the same search, but by property ($expected is untouched):\r
++test_begin_subtest "emacs search by property for both messages"\r
++output=$(notmuch search has:index-decryption=success)\r
++test_expect_equal \\r
++ "$output" \\r
++ "$expected"\r
++\r
++\r
++# try to remove cleartext indexing\r
++test_expect_success 'reindex without cleartext' \\r
++ 'notmuch reindex tag:encrypted and has:index-decryption=success'\r
++test_begin_subtest "reindexed encrypted messages, without cleartext"\r
++output=$(notmuch search wumpus)\r
++expected=''\r
++test_expect_equal \\r
++ "$output" \\r
++ "$expected"\r
++\r
++# and the same search, but by property ($expected is untouched):\r
++test_begin_subtest "emacs search by property with both messages unindexed"\r
++output=$(notmuch search has:index-decryption=success)\r
++test_expect_equal \\r
++ "$output" \\r
++ "$expected"\r
++\r
++# ensure that the tags remain even when we are dropping the cleartext.\r
++test_begin_subtest "verify that tags remain without cleartext"\r
++output=$(notmuch search tag:blarney)\r
++expected='thread:0000000000000004 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox)\r
++thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox)'\r
++test_expect_equal \\r
++ "$output" \\r
++ "$expected"\r
++\r
++\r
++# TODO: test removal of a message from the message store between\r
++# indexing and reindexing.\r
++\r
++# TODO: insert the same message into the message store twice, index,\r
++# remove one of them from the message store, and then reindex.\r
++# reindexing should return a failure but the message should still be\r
++# present? -- or what should the semantics be if you ask to reindex a\r
++# message whose underlying files have been renamed or moved or\r
++# removed?\r
++\r
+ test_done\r
+-- \r
+2.8.1\r
+\r
projects / notmuch-archives.git / commitdiff