Re: [PATCH 3/4] Optimize thread search using matched docid sets.

diff --git a/2a/04729f740a81878623e374f9caa3304cd70b10 b/2a/04729f740a81878623e374f9caa3304cd70b10
new file mode 100644 (file)
index 0000000..9d66e86
--- /dev/null
+++ b/2a/04729f740a81878623e374f9caa3304cd70b10
@@ -0,0 +1,87 @@
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 1D13C40DEFB\r
+       for <notmuch@notmuchmail.org>; Wed, 17 Nov 2010 23:38:42 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -1.9\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5\r
+       tests=[BAYES_00=-1.9] autolearn=ham\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id BRXgZfIOyN53 for <notmuch@notmuchmail.org>;\r
+       Wed, 17 Nov 2010 23:38:32 -0800 (PST)\r
+Received: from dmz-mailsec-scanner-8.mit.edu (DMZ-MAILSEC-SCANNER-8.MIT.EDU\r
+       [18.7.68.37])\r
+       by olra.theworths.org (Postfix) with ESMTP id F1CD540DEF6\r
+       for <notmuch@notmuchmail.org>; Wed, 17 Nov 2010 23:38:31 -0800 (PST)\r
+X-AuditID: 12074425-b7c98ae000000a04-14-4ce4d7f7a7e5\r
+Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])\r
+       by dmz-mailsec-scanner-8.mit.edu (Symantec Brightmail Gateway) with\r
+       SMTP id 09.B7.02564.7F7D4EC4; Thu, 18 Nov 2010 02:38:31 -0500 (EST)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+       by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id oAI7cUnf002393\r
+       for <notmuch@notmuchmail.org>; Thu, 18 Nov 2010 02:38:31 -0500\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+       (authenticated bits=0)\r
+       (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+       by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id oAI7cTEI022928\r
+       (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT)\r
+       for <notmuch@notmuchmail.org>; Thu, 18 Nov 2010 02:38:30 -0500 (EST)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.72)\r
+       (envelope-from <amdragon@mit.edu>) id 1PIz4f-0002Zr-1F\r
+       for notmuch@notmuchmail.org; Thu, 18 Nov 2010 02:38:29 -0500\r
+Date: Thu, 18 Nov 2010 02:38:29 -0500\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: notmuch@notmuchmail.org\r
+Subject: Re: [PATCH 3/4] Optimize thread search using matched docid sets.\r
+Message-ID: <20101118073828.GD2439@mit.edu>\r
+References: <20101117192826.GU2439@mit.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <20101117192826.GU2439@mit.edu>\r
+User-Agent: Mutt/1.5.20 (2009-06-14)\r
+X-Brightmail-Tracker: AAAAAA==\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 18 Nov 2010 07:38:42 -0000\r
+\r
+Currently this code uses a bitmap indexed by docid as a simple, fast\r
+set structure.  This is quite memory-efficient if the docid space is\r
+dense, even if the largest docid is quite large.  Is there a danger\r
+that the docid space will be large and sparse?  Is it worth replacing\r
+this with a smarter bit set structure?\r
+\r
+Quoth myself on Nov 17 at  2:28 pm:\r
+> This reduces thread search's 1+2t Xapian queries (where t is the\r
+> number of matched threads) to 1+t queries and constructs exactly one\r
+> notmuch_message_t for each message instead of 2 to 3.\r
+> notmuch_query_search_threads eagerly fetches the docids of all\r
+> messages matching the user query instead of lazily constructing\r
+> message objects and fetching thread ID's from term lists.\r
+> _notmuch_thread_create takes a seed docid and the set of all matched\r
+> docids and uses a single Xapian query to expand this docid to its\r
+> containing thread, using the matched docid set to determine which\r
+> messages in the thread match the user query instead of using a second\r
+> Xapian query.\r
+> \r
+> As a side effect, this fixes author order so authors are always sorted\r
+> by first occurrence in each thread.  This breaks two emacs tests that\r
+> hard-code the old, buggy author order.\r
+> \r
+> This reduces the amount of time required to load my inbox from 4.523\r
+> seconds to 3.025 seconds (1.5X faster).\r