From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
To: Notmuch Mail <notmuch@notmuchmail.org>
Subject: allow indexing cleartext of encrypted messages
Date: Wed,  9 Dec 2015 22:39:37 -0500
Message-Id: <1449718786-28000-1-git-send-email-dkg@fifthhorseman.net>
Precedence: list

Notmuch currently doesn't index the cleartext of encrypted mail.  This
is the right choice by default, because the index is basically
cleartext-equivalent, and we wouldn't want every indexed mailstore to
leak the contents of its encrypted mails.

However, if a notmuch user has their index in a protected location,
they may prefer the convenience of being able to search the contents
of (at least some of) their encrypted mail.

This series of patches enables notmuch to index the cleartext of
specific encrypted messages when they're being added via "notmuch new"
or "notmuch insert", via a new --try-decrypt flag.

If --try-decrypt is used, and decryption is successful for part of a
message, the message gets an additional "index-decrypted" tag.  If
decryption of part of a message fails, the message gets an additional
"index-decryption-failed" tag.

This tagging approach should allow people to figure out which messages
have been indexed in the clear (or not), and can be used to
selectively reindex them in batch with something like:

----------------
#!/usr/bin/env python3

'''notmuch-reindex.py -- a quick and dirty pythonic mechanism to
re-index specific messages in a notmuch database.  This should
probably be properly implemented as a subcommand for /usr/bin/notmuch
itself'''

import notmuch
import sys

d = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)

query = sys.argv[1]

q = d.create_query(query)

for m in q.search_messages():
    mainfilename = m.get_filename()
    origtags = m.get_tags()
    tags = []
    for t in origtags:
        if t not in ['index-decrypted', 'index-decryption-failed']:
            tags += [t]
    d.begin_atomic()
    for f in m.get_filenames():
        d.remove_message(f)
    (newm,stat) = d.add_message(mainfilename, try_decrypt=True)
    for tag in tags:
        newm.add_tag(tag)
    d.end_atomic()
----------------

A couple key points:

 * There is some code duplication between crypto.c (for the
   notmuch-client) and lib/database.cc and lib/index.cc (for the
   library) because both parts of the codebase use gmime to handle the
   decryption.  I don't want to contaminate the libnotmuch API with
   gmime implementation details, so i don't quite see how to reuse the
   code cleanly.  I'd love suggestions on how to reduce the
   duplications.

 * the libnotmuch API is extended with
   notmuch_database_add_message_try_decrypt().  This should probably
   ultimately be more general, because there are a few additional
   knobs that i can imagine fiddling at indexing time.  For example:

    * verifying cryptographic signatures and storing something about
      those verifications in the notmuch db
     
    * extracting OpenPGP session key information for a given message
      and storing it in a lookaside table in the notmuch db, so that
      it's possible to securely destroy old encryption-capable keys
      and still have local access to the cleartext of the remaining
      messages.

   Some of these additional features might be orthogonal to one
   another as well.  I welcome suggestions for how to improve the API
   so that we don't end up with a combinatorial explosion of
   n_d_add_message_foo() functions.

 * To properly complete this patch series, i think i want to make
   notmuch-reindex.c and add a reindex subcommand, also with a
   --try-decrypt option.  It's not clear to me if the right approach
   for that is to have a C implementation of the python script above
   without modifying libnotmuch, or if i should start by creating a
   notmuch_message_reindex function in libnotmuch, with a try_decrypt
   flag.  Again, suggestions welcome.

 * Is the tagging approach the right thing to do to record success or
   failure of decryption at index time?  Is there a better approach?