--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id B0F4E431FB6\r
+ for <notmuch@notmuchmail.org>; Sat, 28 Jan 2012 10:34:28 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id oESKNv6pQEQY for <notmuch@notmuchmail.org>;\r
+ Sat, 28 Jan 2012 10:34:27 -0800 (PST)\r
+Received: from dmz-mailsec-scanner-6.mit.edu (DMZ-MAILSEC-SCANNER-6.MIT.EDU\r
+ [18.7.68.35])\r
+ by olra.theworths.org (Postfix) with ESMTP id AA998431FAE\r
+ for <notmuch@notmuchmail.org>; Sat, 28 Jan 2012 10:34:27 -0800 (PST)\r
+X-AuditID: 12074423-b7f9c6d0000008c3-a4-4f243fb20c29\r
+Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
+ by dmz-mailsec-scanner-6.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id B6.67.02243.2BF342F4; Sat, 28 Jan 2012 13:34:26 -0500 (EST)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id q0SIYPMe026548; \r
+ Sat, 28 Jan 2012 13:34:26 -0500\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q0SIYOcQ001042\r
+ (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+ Sat, 28 Jan 2012 13:34:25 -0500 (EST)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1RrD5o-0006MC-8C; Sat, 28 Jan 2012 13:33:40 -0500\r
+Date: Sat, 28 Jan 2012 13:33:40 -0500\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Mark Walters <markwalters1009@gmail.com>\r
+Subject: Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag\r
+Message-ID: <20120128183340.GD17991@mit.edu>\r
+References: <20120124011609.GX16740@mit.edu>\r
+ <1327367923-18228-2-git-send-email-markwalters1009@gmail.com>\r
+ <20120124024521.GY16740@mit.edu> <874nvg6qxn.fsf@qmul.ac.uk>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <874nvg6qxn.fsf@qmul.ac.uk>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFmphleLIzCtJLcpLzFFi42IR4hTV1t1kr+JvMHu1mMXquTwW12/OZHZg\r
+ 8tg56y67x7NVt5gDmKK4bFJSczLLUov07RK4Mo7feclSsNmwYsXBF8wNjLfUuhg5OSQETCTm\r
+ da9hg7DFJC7cWw9kc3EICexjlJj58wc7hLOBUWLPow1MEM5JJokHW/ezQjhLGCVuTDvFAtLP\r
+ IqAq8eDbbFYQm01AQ2Lb/uWMILaIgI7E7UML2EFsZgFpiW+/m5lAbGEBZ4nnTV/B6nmBatbc\r
+ WAC1ey2jxKenP9ghEoISJ2c+YYFo1pK48e8lUDMH2KDl/zhAwpxAu14uXgg2R1RARWLKyW1s\r
+ ExiFZiHpnoWkexZC9wJG5lWMsim5Vbq5iZk5xanJusXJiXl5qUW6Znq5mSV6qSmlmxhBgc3u\r
+ oryD8c9BpUOMAhyMSjy8F14p+QuxJpYVV+YeYpTkYFIS5T1rq+IvxJeUn1KZkVicEV9UmpNa\r
+ fIhRgoNZSYT3gxxQjjclsbIqtSgfJiXNwaIkzquh9c5PSCA9sSQ1OzW1ILUIJivDwaEkwcsM\r
+ jGAhwaLU9NSKtMycEoQ0EwcnyHAeoOGyIDW8xQWJucWZ6RD5U4yKUuK8j+2AEgIgiYzSPLhe\r
+ WOJ5xSgO9Iow7w+QKh5g0oLrfgU0mAlocMRVRZDBJYkIKakGRt51fB84szKa9nGt82NRLlfY\r
+ 3CT0nknN+mIIe/SLlyybeSUO8uq/W8mfr7fWO+7i3nBX7bKS+Vla4v97pzw3aq3Ke39k0aNH\r
+ XM8Wn9g44Wa8InNrVTHzvt356/ZvkG3km3QhNnjmn7r7qpfWpywKvJrN4nn7hrRuxQVxiR9J\r
+ p3bE/llYdHbCdiWW4oxEQy3mouJEACnVGSIXAwAA\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sat, 28 Jan 2012 18:34:28 -0000\r
+\r
+Quoth Mark Walters on Jan 28 at 10:51 am:\r
+> \r
+> > > exclude_query = _notmuch_exclude_tags (query, final_query);\r
+> > > \r
+> > > - final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,\r
+> > > - final_query, exclude_query);\r
+> > > + enquire.set_weighting_scheme (Xapian::BoolWeight());\r
+> > > + enquire.set_query (exclude_query);\r
+> > > +\r
+> > > + mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());\r
+> > > +\r
+> > > + GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));\r
+> > > +\r
+> > > + for (iterator = mset.begin (); iterator != mset.end (); iterator++)\r
+> > > + {\r
+> > > + unsigned int doc_id = *iterator;\r
+> > > + g_array_append_val (excluded_doc_ids, doc_id);\r
+> > > + }\r
+> > > + messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);\r
+> > > + _notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,\r
+> > > + excluded_doc_ids);\r
+> > \r
+> > This might be inefficient for message-only queries, since it will\r
+> > fetch *all* excluded docids. This highlights a basic difference\r
+> > between message and thread search: thread search can return messages\r
+> > that don't match the original query and hence needs to know all\r
+> > potentially excluded messages, while message search can only return\r
+> > messages that match the original query.\r
+> \r
+> I now have some benchmarks (not run enough times to be hugely accurate\r
+> so ignore minor differences). The full results are below. The summary\r
+> is:\r
+> \r
+> Large-archive = 1 100 000 messages in 290 000 threads (about 10 years of\r
+> lkml). I mark 1 000 000 deleted\r
+> Small-archive = 70 000 messages in 35 000 threads. 10 000 marked\r
+> deleted.\r
+> \r
+> Doing the initial exclude work on the big collection takes about 0.8s\r
+> and on the small collection about 0.01s. So any query to the big\r
+> collection takes at least 0.8s longer and this all occurs before any\r
+> results appear.\r
+\r
+Interesting. Do you know where that time is spent?\r
+\r
+Also, it might be reasonable to assume that no more than, say, 10% of\r
+a person's mail store is excluded, but maybe that depends on how\r
+people use this feature.\r
+\r
+> I then implemented the exclude doing it once for each thread query in\r
+> _notmuch_create_thread. Roughly this made any query 50% slower.\r
+\r
+That's not terrible.\r
+\r
+> In normal front end use even the 0.8s is not totally unusable, but it is\r
+> totally unacceptable in the backend where a user might do something like\r
+> \r
+> for i in ` notmuch search --output=threads from:xxx ` ; \r
+> do \r
+> notmuch search --output=messages $i; \r
+> done\r
+> \r
+> to list all messages in all matching threads.\r
+> \r
+> So I think my conclusions are:\r
+> \r
+> (1) message only queries must be done without the full exclude.\r
+> (2) thread queries which only match one message should not do the full\r
+> exclude\r
+> (3) it would be nice to switch between the two approaches depending on\r
+> size but I don't see how to do that without extra(!) queries\r
+> (4) One possible might be do something that say does thirty threads with\r
+> the by thread method and then if not finished does the full exclude.\r
+> (5) thread-by-thread might be best for Jani's limit-match \r
+> id:"1327692900-22926-1-git-send-email-jani@nikula.org" \r
+> \r
+> Obviously, anything setting an exclude flag like this will be slower\r
+> (since it is doing more work): the question is are either of these (or a\r
+> combination like (4) above) acceptable?\r
+\r
+Or only mark matched messages as excluded.\r
+\r
+Here's another idea (actually, a rehash of an old idea). For message\r
+search do two queries, the original query and "<original> AND\r
+<exclude>", and use this to keep everything in order and mark excluded\r
+messages. For thread search, use message search results so it's easy\r
+to both sort by unexcluded messages and include fully-excluded\r
+threads, but compute the excluded flag (either just for unmatched\r
+messages or for all messages) by examining each message's tags\r
+directly (which thread_add_message already iterates over, so this is\r
+easy and won't add any overhead). If the excluded query is fast,\r
+which I think it will be, I think this should get the best of all\r
+worlds and be fairly straightforward to implement (no asymmetries\r
+between the queries used for message and thread search). It would be\r
+easy and worth it to run the excluded query by hand on your test\r
+corpus; I suspect it will be much faster than 0.8s because the query\r
+already uses "Tmail", which is huge and doesn't seem to slow things\r
+down.\r
+\r
+> I now have a mostly working implementation from library to\r
+> emacs frontend and I do like the overall outcome.\r
+\r
+Awesome.\r
+\r
+> The complete benchmarks are below\r
+> \r
+> Best wishes\r
+> \r
+> Mark\r
+> \r
+> LARGE COLLECTION is 1,100,000 messages 290,000 threads 1,000,000 deleted\r
+> SMALL COLLECTION is 70,000 messages in 35,000 threads 10,000 deleted\r
+> \r
+> benchmarks: all times in seconds, x/y/z means a query which matches x\r
+> threads with y matching messages and z messages in total. Ig or ignore\r
+> means with the tag-exclude turned off (i.e. with a query matching the\r
+> excluded tag). list all messages is the time for the for loop listed\r
+> above giving all message-ids for all messages in any thread matching a\r
+> query.\r
+> \r
+> Finally the three columns are master with exclude code disabled,\r
+> thread-thread is doing excludes once per thread construction, and\r
+> in-advance does all the exclude work in advance as in the patches I posted.\r
+> \r
+> In most cases the benchmark is the average of a lot of runs so the\r
+> database should have been as cached as one could hope.\r
+> \r
+> master-(all) thread-thread in-advance\r
+> LARGE COLLECTION \r
+> show single message 0.016 0.018 0.78\r
+> search single message 0.015 0.016 0.78\r
+> search single with tag 0.015 0.015 0.009\r
+> 945/2627/20000\r
+> query ignore 2.9 n/a 3\r
+> query 2.9 4.2 3.8\r
+> list all messages (ig) 13 n/a 13\r
+> list all messages 13 14 12mins\r
+> 4754/13000/110000\r
+> query ignore 15.9 n/a 17\r
+> query 15.9 22 17.6\r
+> only messages 1.25 1.26 1.9\r
+> 177/483/1752 \r
+> query 0.3 0.42 1.1\r
+> \r
+> search '*' 20mins 28mins 21.5mins \r
+> \r
+> SMALL COLLECTION\r
+> 1500/2800/5600\r
+> query 1.8 2.7 2\r
+> list all messages 14.5 16.4 30\r
+> single message 0.008 0.008 0.018\r
+> \r
+> search '*' 28 49 32\r
+> \r
projects / notmuch-archives.git / commitdiff