FastMail uses xapian for real time search, can we do the same?

diff --git a/64/835bdf3b02d6ecd3c51ae494b5f488bd09a29f b/64/835bdf3b02d6ecd3c51ae494b5f488bd09a29f
new file mode 100644 (file)
index 0000000..4e4065e
--- /dev/null
+++ b/64/835bdf3b02d6ecd3c51ae494b5f488bd09a29f
@@ -0,0 +1,79 @@
+Return-Path: <edward@4angle.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 49C5F6DE00F1\r
+ for <notmuch@notmuchmail.org>; Thu, 24 Mar 2016 03:40:47 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -1.083\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-1.083 tagged_above=-999 required=5\r
+ tests=[AWL=-1.073, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id kMPWFhRXwnlV for <notmuch@notmuchmail.org>;\r
+ Thu, 24 Mar 2016 03:40:38 -0700 (PDT)\r
+Received: from 4angle.com (4angle.com [82.145.46.9])\r
+ by arlo.cworth.org (Postfix) with ESMTPS id B56036DE0005\r
+ for <notmuch@notmuchmail.org>; Thu, 24 Mar 2016 03:40:38 -0700 (PDT)\r
+Received: from cpc91242-cmbg18-2-0-cust1730.5-4.cable.virginm.net\r
+ ([82.8.134.195] helo=x230)\r
+ by 4angle.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)\r
+ (Exim 4.86) (envelope-from <edward@4angle.com>) id 1aj2gr-0008Bj-QV\r
+ for notmuch@notmuchmail.org; Thu, 24 Mar 2016 10:40:33 +0000\r
+Received: from edward by x230 with local (Exim 4.87)\r
+ (envelope-from <edward@x230>) id 1aj2gr-0006sc-8s\r
+ for notmuch@notmuchmail.org; Thu, 24 Mar 2016 10:40:33 +0000\r
+Date: Thu, 24 Mar 2016 10:40:33 +0000\r
+From: Edward Betts <edward@4angle.com>\r
+To: notmuch@notmuchmail.org\r
+Subject: FastMail uses xapian for real time search, can we do the same?\r
+Message-ID: <20160324104033.GA23082@4angle.com>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+User-Agent: Mutt/1.5.24 (2015-08-30)\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 24 Mar 2016 10:40:47 -0000\r
+\r
+I've tried using 'notmuch insert' with mutt-kz, the mutt fork that talks to\r
+the notmuch database. Both keep trying to grab the write lock at the same time.\r
+\r
+I tag a mail in mutt-kz and it writes the new tag to the database, at the same\r
+time a new mail is coming in. Either 'notmuch insert' has the lock and\r
+my mail client pauses while it retries or notmuch-insert isn't able to get the\r
+write lock and just saves the mail to the maildir and it doesn't get added\r
+until I later run 'notmuch new'.\r
+\r
+I guess many people here are using the Emacs client and the problem is partly\r
+avoided because there is a separate thread in the background trying to grab\r
+the write lock and save tag changes. Is that correct?\r
+\r
+I wonder if the solution is real time search? xapian can search in multiple\r
+databases. New mails and tag changes can be written to a new database, there\r
+could be one database for inserting new mails and another for changes to the\r
+tags on existing mails. Then there are no problems with write locks. These\r
+databases will be much smaller, so the writes should be faster. There can be a\r
+process that runs once a day to merge the databases.\r
+\r
+FastMail have implemented real time search for email with xapian.\r
+\r
+Blog post: https://blog.fastmail.com/2014/12/01/email-search-system/\r
+\r
+Implementation: https://github.com/brong/cyrus-imapd/tree/fastmail\r
+\r
+Is there any enthusiasm for adding real time search to notmuch?\r
+-- \r
+Edward.\r