From: Edward Betts Date: Thu, 24 Mar 2016 10:40:33 +0000 (+0000) Subject: FastMail uses xapian for real time search, can we do the same? X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=dcd4452fb0ab244fab38dd4c7cc633bc65ba44e9;p=notmuch-archives.git FastMail uses xapian for real time search, can we do the same? --- diff --git a/64/835bdf3b02d6ecd3c51ae494b5f488bd09a29f b/64/835bdf3b02d6ecd3c51ae494b5f488bd09a29f new file mode 100644 index 000000000..4e4065e25 --- /dev/null +++ b/64/835bdf3b02d6ecd3c51ae494b5f488bd09a29f @@ -0,0 +1,79 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by arlo.cworth.org (Postfix) with ESMTP id 49C5F6DE00F1 + for ; Thu, 24 Mar 2016 03:40:47 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at cworth.org +X-Spam-Flag: NO +X-Spam-Score: -1.083 +X-Spam-Level: +X-Spam-Status: No, score=-1.083 tagged_above=-999 required=5 + tests=[AWL=-1.073, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled +Received: from arlo.cworth.org ([127.0.0.1]) + by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id kMPWFhRXwnlV for ; + Thu, 24 Mar 2016 03:40:38 -0700 (PDT) +Received: from 4angle.com (4angle.com [82.145.46.9]) + by arlo.cworth.org (Postfix) with ESMTPS id B56036DE0005 + for ; Thu, 24 Mar 2016 03:40:38 -0700 (PDT) +Received: from cpc91242-cmbg18-2-0-cust1730.5-4.cable.virginm.net + ([82.8.134.195] helo=x230) + by 4angle.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) + (Exim 4.86) (envelope-from ) id 1aj2gr-0008Bj-QV + for notmuch@notmuchmail.org; Thu, 24 Mar 2016 10:40:33 +0000 +Received: from edward by x230 with local (Exim 4.87) + (envelope-from ) id 1aj2gr-0006sc-8s + for notmuch@notmuchmail.org; Thu, 24 Mar 2016 10:40:33 +0000 +Date: Thu, 24 Mar 2016 10:40:33 +0000 +From: Edward Betts +To: notmuch@notmuchmail.org +Subject: FastMail uses xapian for real time search, can we do the same? +Message-ID: <20160324104033.GA23082@4angle.com> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.5.24 (2015-08-30) +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.20 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 24 Mar 2016 10:40:47 -0000 + +I've tried using 'notmuch insert' with mutt-kz, the mutt fork that talks to +the notmuch database. Both keep trying to grab the write lock at the same time. + +I tag a mail in mutt-kz and it writes the new tag to the database, at the same +time a new mail is coming in. Either 'notmuch insert' has the lock and +my mail client pauses while it retries or notmuch-insert isn't able to get the +write lock and just saves the mail to the maildir and it doesn't get added +until I later run 'notmuch new'. + +I guess many people here are using the Emacs client and the problem is partly +avoided because there is a separate thread in the background trying to grab +the write lock and save tag changes. Is that correct? + +I wonder if the solution is real time search? xapian can search in multiple +databases. New mails and tag changes can be written to a new database, there +could be one database for inserting new mails and another for changes to the +tags on existing mails. Then there are no problems with write locks. These +databases will be much smaller, so the writes should be faster. There can be a +process that runs once a day to merge the databases. + +FastMail have implemented real time search for email with xapian. + +Blog post: https://blog.fastmail.com/2014/12/01/email-search-system/ + +Implementation: https://github.com/brong/cyrus-imapd/tree/fastmail + +Is there any enthusiasm for adding real time search to notmuch? +-- +Edward.