--- /dev/null
+Return-Path: <dkg@fifthhorseman.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id B6F216DE1601\r
+ for <notmuch@notmuchmail.org>; Wed, 9 Dec 2015 19:40:13 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.033\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.033 tagged_above=-999 required=5\r
+ tests=[AWL=-0.033] autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id Gxu-qmnTqJ1x for <notmuch@notmuchmail.org>;\r
+ Wed, 9 Dec 2015 19:40:12 -0800 (PST)\r
+Received: from che.mayfirst.org (che.mayfirst.org [209.234.253.108])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 094F26DE1830\r
+ for <notmuch@notmuchmail.org>; Wed, 9 Dec 2015 19:40:06 -0800 (PST)\r
+Received: from fifthhorseman.net (unknown [38.109.115.130])\r
+ by che.mayfirst.org (Postfix) with ESMTPSA id 21197F984\r
+ for <notmuch@notmuchmail.org>; Wed, 9 Dec 2015 22:40:03 -0500 (EST)\r
+Received: by fifthhorseman.net (Postfix, from userid 1000)\r
+ id 9D1AB20548; Wed, 9 Dec 2015 22:40:03 -0500 (EST)\r
+From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>\r
+To: Notmuch Mail <notmuch@notmuchmail.org>\r
+Subject: allow indexing cleartext of encrypted messages\r
+Date: Wed, 9 Dec 2015 22:39:37 -0500\r
+Message-Id: <1449718786-28000-1-git-send-email-dkg@fifthhorseman.net>\r
+X-Mailer: git-send-email 2.6.2\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 10 Dec 2015 03:40:14 -0000\r
+\r
+Notmuch currently doesn't index the cleartext of encrypted mail. This\r
+is the right choice by default, because the index is basically\r
+cleartext-equivalent, and we wouldn't want every indexed mailstore to\r
+leak the contents of its encrypted mails.\r
+\r
+However, if a notmuch user has their index in a protected location,\r
+they may prefer the convenience of being able to search the contents\r
+of (at least some of) their encrypted mail.\r
+\r
+This series of patches enables notmuch to index the cleartext of\r
+specific encrypted messages when they're being added via "notmuch new"\r
+or "notmuch insert", via a new --try-decrypt flag.\r
+\r
+If --try-decrypt is used, and decryption is successful for part of a\r
+message, the message gets an additional "index-decrypted" tag. If\r
+decryption of part of a message fails, the message gets an additional\r
+"index-decryption-failed" tag.\r
+\r
+This tagging approach should allow people to figure out which messages\r
+have been indexed in the clear (or not), and can be used to\r
+selectively reindex them in batch with something like:\r
+\r
+----------------\r
+#!/usr/bin/env python3\r
+\r
+'''notmuch-reindex.py -- a quick and dirty pythonic mechanism to\r
+re-index specific messages in a notmuch database. This should\r
+probably be properly implemented as a subcommand for /usr/bin/notmuch\r
+itself'''\r
+\r
+import notmuch\r
+import sys\r
+\r
+d = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)\r
+\r
+query = sys.argv[1]\r
+\r
+q = d.create_query(query)\r
+\r
+for m in q.search_messages():\r
+ mainfilename = m.get_filename()\r
+ origtags = m.get_tags()\r
+ tags = []\r
+ for t in origtags:\r
+ if t not in ['index-decrypted', 'index-decryption-failed']:\r
+ tags += [t]\r
+ d.begin_atomic()\r
+ for f in m.get_filenames():\r
+ d.remove_message(f)\r
+ (newm,stat) = d.add_message(mainfilename, try_decrypt=True)\r
+ for tag in tags:\r
+ newm.add_tag(tag)\r
+ d.end_atomic()\r
+----------------\r
+\r
+A couple key points:\r
+\r
+ * There is some code duplication between crypto.c (for the\r
+ notmuch-client) and lib/database.cc and lib/index.cc (for the\r
+ library) because both parts of the codebase use gmime to handle the\r
+ decryption. I don't want to contaminate the libnotmuch API with\r
+ gmime implementation details, so i don't quite see how to reuse the\r
+ code cleanly. I'd love suggestions on how to reduce the\r
+ duplications.\r
+\r
+ * the libnotmuch API is extended with\r
+ notmuch_database_add_message_try_decrypt(). This should probably\r
+ ultimately be more general, because there are a few additional\r
+ knobs that i can imagine fiddling at indexing time. For example:\r
+\r
+ * verifying cryptographic signatures and storing something about\r
+ those verifications in the notmuch db\r
+ \r
+ * extracting OpenPGP session key information for a given message\r
+ and storing it in a lookaside table in the notmuch db, so that\r
+ it's possible to securely destroy old encryption-capable keys\r
+ and still have local access to the cleartext of the remaining\r
+ messages.\r
+\r
+ Some of these additional features might be orthogonal to one\r
+ another as well. I welcome suggestions for how to improve the API\r
+ so that we don't end up with a combinatorial explosion of\r
+ n_d_add_message_foo() functions.\r
+\r
+ * To properly complete this patch series, i think i want to make\r
+ notmuch-reindex.c and add a reindex subcommand, also with a\r
+ --try-decrypt option. It's not clear to me if the right approach\r
+ for that is to have a C implementation of the python script above\r
+ without modifying libnotmuch, or if i should start by creating a\r
+ notmuch_message_reindex function in libnotmuch, with a try_decrypt\r
+ flag. Again, suggestions welcome.\r
+\r
+ * Is the tagging approach the right thing to do to record success or\r
+ failure of decryption at index time? Is there a better approach?\r
+\r
+\r
projects / notmuch-archives.git / commitdiff