Re: [notmuch] Rather simple optimization for notmuch tag

diff --git a/45/f15ec42eac65707d07c5544ae2f486623b16d9 b/45/f15ec42eac65707d07c5544ae2f486623b16d9
new file mode 100644 (file)
index 0000000..9112edc
--- /dev/null
+++ b/45/f15ec42eac65707d07c5544ae2f486623b16d9
@@ -0,0 +1,107 @@
+Return-Path: <gmn-notmuch@m.gmane.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id A7C81431FBC\r
+       for <notmuch@notmuchmail.org>; Wed, 23 Dec 2009 16:27:31 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id C2UDU+sN-uGv for <notmuch@notmuchmail.org>;\r
+       Wed, 23 Dec 2009 16:27:28 -0800 (PST)\r
+Received: from lo.gmane.org (lo.gmane.org [80.91.229.12])\r
+       by olra.theworths.org (Postfix) with ESMTP id 825EE431FAE\r
+       for <notmuch@notmuchmail.org>; Wed, 23 Dec 2009 16:27:28 -0800 (PST)\r
+Received: from list by lo.gmane.org with local (Exim 4.50) id 1NNbXx-0000D6-Tr\r
+       for notmuch@notmuchmail.org; Thu, 24 Dec 2009 01:27:18 +0100\r
+Received: from ip-118-90-130-94.xdsl.xnet.co.nz ([118.90.130.94])\r
+       by main.gmane.org with esmtp (Gmexim 0.1 (Debian))\r
+       id 1AlnuQ-0007hv-00\r
+       for <notmuch@notmuchmail.org>; Thu, 24 Dec 2009 01:27:17 +0100\r
+Received: from olly by ip-118-90-130-94.xdsl.xnet.co.nz with local (Gmexim 0.1\r
+       (Debian)) id 1AlnuQ-0007hv-00\r
+       for <notmuch@notmuchmail.org>; Thu, 24 Dec 2009 01:27:17 +0100\r
+X-Injected-Via-Gmane: http://gmane.org/\r
+To: notmuch@notmuchmail.org\r
+From: Olly Betts <olly@survex.com>\r
+Date: Thu, 24 Dec 2009 00:27:00 +0000 (UTC)\r
+Lines: 49\r
+Message-ID: <loom.20091224T002659-835@post.gmane.org>\r
+References: <3wdskb8oh77.fsf@testarossa.amd.com>\r
+       <87hbroyyf6.fsf@yoom.home.cworth.org>\r
+       <loom.20091223T043223-941@post.gmane.org>\r
+       <3wd637xo8oq.fsf@testarossa.amd.com>\r
+Mime-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Transfer-Encoding: 7bit\r
+X-Complaints-To: usenet@ger.gmane.org\r
+X-Gmane-NNTP-Posting-Host: sea.gmane.org\r
+User-Agent: Loom/3.14 (http://gmane.org/)\r
+X-Loom-IP: 118.90.130.94 (Mozilla/5.0 (X11; U; Linux x86_64; en-GB;\r
+       rv:1.9.1.6) Gecko/20091215 Ubuntu/9.10 (karmic) Firefox/3.5.6)\r
+Sender: news <news@ger.gmane.org>\r
+Subject: Re: [notmuch] Rather simple optimization for notmuch tag\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.12\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 24 Dec 2009 00:27:31 -0000\r
+\r
+Mark Anderson writes:\r
+> On Wed, 23 Dec 2009 03:45:14 +0000, Olly Betts wrote:\r
+> > Handling a combination of removals and additions is trickier, but probably\r
+> > possible, although the more tags you are dealing with, the less profitable\r
+> > the filtering is likely to be (as the filter is likely to cull fewer\r
+> > documents yet be more expensive to evaluate).\r
+> \r
+> But the transform is pretty simple, I think that any combination of\r
+> additions and removals could be transformed according to the following\r
+> formula.\r
+> \r
+> notmuch tag +a1 +a2 +a3 -d1 -d2 -d3 <search-terms>\r
+> \r
+> would transform to something like:\r
+> \r
+> <search-terms> and ( not(a1) or not(a2) or not(a3) or d1 or d2 or d3)\r
+\r
+Note that Xapian doesn't really have a "not" operator (because of how it\r
+works - by storing the documents indexing each term - rather than because\r
+nobody's implemented it), so it isn't quite as simple as the above.\r
+\r
+There is a posting list for "all documents" (which is very efficient if\r
+the document ids form a contiguous range; if they don't, it's as efficient\r
+as a term which matches all those documents for the chert backend, but not\r
+so great for the default flint backend in 1.0.x), and you can combine this\r
+with the "AND_NOT" operator to give the equivalent of a "NOT" operator.\r
+\r
+So I think the example above is probably best expressed as:\r
+\r
+( <search-terms> AND ( ( ALL AND_NOT (a1 AND a2 AND a3) ) OR d1 OR d2 OR d3 )\r
+\r
+But my point wasn't that I doubted it could be handled, but that it becomes\r
+less worthwhile as the number of tags increases (and at some point will\r
+become slower).\r
+\r
+> There are certainly may be much more optimal ways to do it depending on\r
+> the specific corpus of the database, considering if the tags a1 and a2\r
+> and a3 are usually added as one tag, or if the addition is done\r
+> individually, because if I know that a3 implies a1 and a2, the first 3\r
+> terms could be combined to not(a1 and a2 and a3), or I could just\r
+> exclude a3 tagged messages for nearly the same effect, with expected\r
+> performance improvements.\r
+\r
+I think you always can combine them like that.  The documents that don't\r
+need looking at are precisely those which already have all three tags\r
+(i.e. a1 AND a2 AND a3), so those that do are "NOT" that expression.\r
+\r
+Cheers,\r
+    Olly\r
+\r