Re: [PATCH v4 16/16] add "notmuch reindex" subcommand

diff --git a/16/b4a247fb7c5750b4fa3be50ee8eb3394515a47 b/16/b4a247fb7c5750b4fa3be50ee8eb3394515a47
new file mode 100644 (file)
index 0000000..f69f1de
--- /dev/null
+++ b/16/b4a247fb7c5750b4fa3be50ee8eb3394515a47
@@ -0,0 +1,96 @@
+Return-Path: <david@tethera.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 03D096DE8256\r
+ for <notmuch@notmuchmail.org>; Sun, 14 Aug 2016 16:41:58 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.007\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.007 tagged_above=-999 required=5 tests=[AWL=0.004,\r
+  SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id RDVhMFu7Q6Lb for <notmuch@notmuchmail.org>;\r
+ Sun, 14 Aug 2016 16:41:49 -0700 (PDT)\r
+Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197])\r
+ by arlo.cworth.org (Postfix) with ESMTPS id CA20E6DEAEE1\r
+ for <notmuch@notmuchmail.org>; Sun, 14 Aug 2016 16:24:10 -0700 (PDT)\r
+Received: from remotemail by fethera.tethera.net with local (Exim 4.84_2)\r
+ (envelope-from <david@tethera.net>)\r
+ id 1bZ4lM-0005JT-Ao; Sun, 14 Aug 2016 19:24:16 -0400\r
+Received: (nullmailer pid 13226 invoked by uid 1000);\r
+ Sun, 14 Aug 2016 22:42:39 -0000\r
+From: David Bremner <david@tethera.net>\r
+To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>,\r
+ Notmuch Mail <notmuch@notmuchmail.org>\r
+Cc: Olly Betts <olly@survex.com>\r
+Subject: Re: [PATCH v4 16/16] add "notmuch reindex" subcommand\r
+In-Reply-To: <1467970047-8013-17-git-send-email-dkg@fifthhorseman.net>\r
+References: <1467970047-8013-1-git-send-email-dkg@fifthhorseman.net>\r
+ <1467970047-8013-17-git-send-email-dkg@fifthhorseman.net>\r
+User-Agent: Notmuch/0.22.1+61~g2ce0f13 (https://notmuchmail.org) Emacs/24.5.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Mon, 15 Aug 2016 07:42:39 +0900\r
+Message-ID: <87h9an3rc0.fsf@maritornes.cs.unb.ca>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 14 Aug 2016 23:41:58 -0000\r
+\r
+Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:\r
+\r
+\r
+> +Supported options for **reindex** include\r
+> +\r
+> +    ``--try-decrypt``\r
+> +\r
+> +        For each message, if it is encrypted, try to decrypt it while\r
+> +        indexing.  If decryption is successful, index the cleartext\r
+> +        itself.  Be aware that the index is likely sufficient to\r
+> +        reconstruct the cleartext of the message itself, so please\r
+> +        ensure that the notmuch message index is adequately\r
+> +        protected. DO NOT USE THIS FLAG without considering the\r
+> +        security of your index.\r
+\r
+What can we say about re-indexing without the flag, when the user has\r
+previously indexed cleartext? I guess this is at least partly a question\r
+for Olly: if we delete terms from a xapian document, how recoverable are\r
+those terms and  positions? I suppose it might depend on backend, but\r
+does deleting terms provide at least same level of security as deleting\r
+files in modern file systems (i.e. not much against determined state\r
+level actors, but good enough to defeat most older brothers)\r
+\r
+> +# TODO: test removal of a message from the message store between\r
+> +# indexing and reindexing.\r
+> +\r
+> +# TODO: insert the same message into the message store twice, index,\r
+> +# remove one of them from the message store, and then reindex.\r
+> +# reindexing should return a failure but the message should still be\r
+> +# present? -- or what should the semantics be if you ask to reindex a\r
+> +# message whose underlying files have been renamed or moved or\r
+> +# removed?\r
+\r
+These tests don't seem hard to impliment, just a bit of drudge work?\r
+\r
+TBH I'd have to read source to figure out the degree of robustness\r
+promised by e.g. show/search for renames (without intervening new).\r
+There is some argument for reindexing by path as being a useful use\r
+case, if it could handle renames.\r
+\r
+In any case, it would be nice (TM) to document what the current\r
+behaviour is for users.\r
+\r
+d\r