--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 0CB84431FD0\r
+ for <notmuch@notmuchmail.org>; Mon, 30 May 2011 18:05:26 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id rXU1Lv0vXik0 for <notmuch@notmuchmail.org>;\r
+ Mon, 30 May 2011 18:05:25 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-6.mit.edu (DMZ-MAILSEC-SCANNER-6.MIT.EDU\r
+ [18.7.68.35])\r
+ by olra.theworths.org (Postfix) with ESMTP id DFBF2431FB6\r
+ for <notmuch@notmuchmail.org>; Mon, 30 May 2011 18:05:24 -0700 (PDT)\r
+X-AuditID: 12074423-b7babae000007c6b-8a-4de43ecbbdd4\r
+Received: from mailhub-auth-2.mit.edu ( [18.7.62.36])\r
+ by dmz-mailsec-scanner-6.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 08.97.31851.BCE34ED4; Mon, 30 May 2011 21:05:15 -0400 (EDT)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id p4V15NKp006607; \r
+ Mon, 30 May 2011 21:05:23 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id p4V15LlI011545\r
+ (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+ Mon, 30 May 2011 21:05:22 -0400 (EDT)\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.72)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1QRDOV-0006tm-L9; Mon, 30 May 2011 21:05:15 -0400\r
+Date: Mon, 30 May 2011 21:05:15 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Patrick Totzke <patricktotzke@googlemail.com>\r
+Subject: Re: one-time-iterators\r
+Message-ID: <20110531010515.GW29861@mit.edu>\r
+References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org>\r
+ <BANLkTi=3mQYJft4s9jGaoqSbcJvqhmZXyQ@mail.gmail.com>\r
+ <1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu>\r
+ <1306446621-sup-3184@brick>\r
+ <BANLkTi=Uk+bNB8sCZLVb86q-Kjfx1udEZA@mail.gmail.com>\r
+ <1306518628-sup-5396@brick>\r
+ <BANLkTi=cZ50h50xf_OigTyjdfY_y34AX_g@mail.gmail.com>\r
+ <1306572273-sup-9995@brick>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=iso-8859-1\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: 8bit\r
+In-Reply-To: <1306572273-sup-9995@brick>\r
+User-Agent: Mutt/1.5.20 (2009-06-14)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFlrJKsWRmVeSWpSXmKPExsUixG6nonva7omvwcEVlhbXb85ktng28QO7\r
+ A5PH0wmT2T2erbrFHMAUxWWTkpqTWZZapG+XwJXxu3M1U8Ehw4opb5+xNTC2q3cxcnJICJhI\r
+ fHn1kR3CFpO4cG89WxcjF4eQwD5GiSvLX7NDOBsYJeZMamKFcE4ySZw508cE4SxhlLj0YT4T\r
+ SD+LgKrEs+ZbYDabgIbEtv3LGUFsEQFDib7ZB9hAbGagmsa1F5lBbGEBeYm1LbNYQWxeAR2J\r
+ 3Q9OQa3rZ5b4eeUrG0RCUOLkzCcsEM06Eju33gGKcwDZ0hLL/3FAhOUlmrfOBpvJCbT3+/+/\r
+ YOWiAioS1/a3s01gFJ6FZNIsJJNmIUyahWTSAkaWVYyyKblVurmJmTnFqcm6xcmJeXmpRbpm\r
+ ermZJXqpKaWbGMGx4KK8g/HPQaVDjAIcjEo8vNz7H/sKsSaWFVfmHmKU5GBSEuU9bvvEV4gv\r
+ KT+lMiOxOCO+qDQntfgQowQHs5II7zc+oBxvSmJlVWpRPkxKmoNFSZx3rqS6r5BAemJJanZq\r
+ akFqEUxWhoNDSYL3DMhQwaLU9NSKtMycEoQ0EwcnyHAeoOGudiDDiwsSc4sz0yHypxh1Oa6v\r
+ 3XqQUYglLz8vVUqcVwakSACkKKM0D24OLIW9YhQHekuYlwWkigeY/uAmvQJawgS0pPfdQ5Al\r
+ JYkIKakGRjVl0yOa3ftTH32Q9FQ0OHvW/uDFaQc0HZ1mz2Nl2rHpxhSVkx+yp5rYbDPaGv5Y\r
+ uc95SuLLfgEV08yt9/gf5O/snzJl01clH++YzU3pPs46txiaeB/qfLbdJOPkvfFppkmk7qbl\r
+ T71X/M3v2BC+lvfkf901sswKu2cJNydlbny4IDnu6l2u60osxRmJhlrMRcWJACjtOP88AwAA\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 31 May 2011 01:05:26 -0000\r
+\r
+Quoth Patrick Totzke on May 28 at 9:58 am:\r
+> Excerpts from Austin Clements's message of Fri May 27 20:29:24 +0100 2011:\r
+> > On Fri, May 27, 2011 at 2:04 PM, Patrick Totzke\r
+> > <patricktotzke@googlemail.com> wrote:\r
+> > > Excerpts from Austin Clements's message of Fri May 27 03:41:44 +0100 2011:\r
+> > >> >> > > Have you tried simply calling list() on your thread\r
+> > >> >> > > iterator to see how expensive it is? My bet is that it's quite cheap,\r
+> > >> >> > > both memory-wise and CPU-wise.\r
+> > >> >> > Funny thing:\r
+> > >> >> > q=Database().create_query('*')\r
+> > >> >> > time tlist = list(q.search_threads())\r
+> > >> >> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some reason\r
+> > >> >> > the list constructor must read mere than once from the iterator.\r
+> > >> >> > So this is not an option, but even if it worked, it would show\r
+> > >> >> > the same behaviour as my above test..\r
+> > >> >>\r
+> > >> >> Interesting. Looks like the Threads class implements __len__ and that\r
+> > >> >> its implementation exhausts the iterator. Which isn't a great idea in\r
+> > >> >> itself, but it turns out that Python's implementation of list() calls\r
+> > >> >> __len__ if it's available (presumably to pre-size the list) before\r
+> > >> >> iterating over the object, so it exhausts the iterator before even\r
+> > >> >> using it.\r
+> > >> >>\r
+> > >> >> That said, if list(q.search_threads()) did work, it wouldn't give you\r
+> > >> >> better performance than your experiment above.\r
+> > > true. Nevertheless I think that list(q.search_threads())\r
+> > > should be equivalent to [t for t in q.search_threads()], which is\r
+> > > something to be fixed in the bindings. Should I file an issue somehow?\r
+> > > Or is enough to state this as a TODO here on the list?\r
+> > \r
+> > Yes, they should be equivalent.\r
+> > \r
+> > Sebastian was thinking about fixing the larger issue of generator\r
+> > exhaustion, which would address this, though the performance would\r
+> > depend on the cost of iterating twice. This is why generators\r
+> > shouldn't support __len__. Unfortunately, it's probably hard to get\r
+> > rid of at this point and I doubt there's a way to tell list() to\r
+> > overlook the presence of a __len__ method.\r
+> Why not simply removing support for __len__ in the Threads and Messages classes?\r
+\r
+Presumably len is there because things use it. On the other hand,\r
+given the issues surrounding len, I suspect anything that's using it\r
+is already a mess.\r
+\r
+> > >> >> > would it be very hard to implement a Query.search_thread_ids() ?\r
+> > >> >> > This name is a bit off because it had to be done on a lower level.\r
+> > >> >>\r
+> > >> >> Lazily fetching the thread metadata on the C side would probably\r
+> > >> >> address your problem automatically. But what are you doing that\r
+> > >> >> doesn't require any information about the threads you're manipulating?\r
+> > >> > Agreed. Unfortunately, there seems to be no way to get a list of thread\r
+> > >> > ids or a reliable iterator thereof by using the current python bindings.\r
+> > >> > It would be enough for me to have the ids because then I could\r
+> > >> > search for the few threads I actually need individually on demand.\r
+> > >>\r
+> > >> There's no way to do that from the C API either, so don't feel left\r
+> > >> out. ]:--8) It seems to me that the right solution to your problem\r
+> > >> is to make thread information lazy (effectively, everything gathered\r
+> > >> in lib/thread.cc:_thread_add_message). Then you could probably\r
+> > >> materialize that iterator cheaply.\r
+> > > Alright. I'll put this on my mental notmuch wish list and\r
+> > > hope that someone will have addressed this before I run out of\r
+> > > ideas how to improve my UI and have time to look at this myself.\r
+> > > For now, I go with the [t.get_thread_id for t in q.search_threads()]\r
+> > > approach to cache the thread ids myself and live with the fact that\r
+> > > this takes time for large result sets.\r
+> > >\r
+> > >> In fact, it's probably worth\r
+> > >> trying a hack where you put dummy information in the thread object\r
+> > >> from _thread_add_message and see how long it takes just to walk the\r
+> > >> iterator (unfortunately I don't think profiling will help much here\r
+> > >> because much of your time is probably spent waiting for I/O).\r
+> > > I don't think I understand what you mean by dummy info in a thread\r
+> > > object.\r
+> > \r
+> > In _thread_add_message, rather than looking up the message's author,\r
+> > subject, etc, just hard-code some dummy values. Performance-wise,\r
+> > this would simulate making the thread metadata lookup lazy, so you\r
+> > could see if making this lazy would address your problem.\r
+> Thanks for the clarification. I did that, and also commented out the\r
+> lower parts of _notmuch_thread_create and this did indeed improve\r
+> the performance, but not so much as I had hoped:\r
+> \r
+> In [10]: q=Database().create_query('*')\r
+> In [11]: time T=[t for t in q.search_threads()]\r
+> CPU times: user 2.43 s, sys: 0.22 s, total: 2.65 s\r
+> Wall time: 2.66 s\r
+> \r
+> And I have only about 8000 mails in my index.\r
+> Making thread lookups lazy would help, but here one would still\r
+> create a lot of unused (empty) thread objects.\r
+> The easiest solution to my problem would in my opinion be\r
+> a function that queries only for thread ids without instanciating them.\r
+> But I can't think of any other use case than mine for this\r
+> so I guess many of you would be against adding this to the API?\r
+\r
+Well, you may be in a pickle. I seriously doubt creating the thread\r
+objects is your problem, though *expanding* the threads may be your\r
+main expense now. I'd recommend actual profiling at this point.\r
+\r
+It may be possible to lazily expand threads, too. I'm not sure how\r
+familiar you are with the C code for this. notmuch does *not* index\r
+threads; it indexes individual messages. When you do a thread query,\r
+it finds all *messages* that match that query, and then, for each\r
+message, it does another query to find all messages in the same\r
+thread. You might be able to do just the original message query\r
+eagerly (which takes virtually no time) and then do the thread\r
+expansion queries only as the iterator demands them. You may\r
+encounter concurrency issues with a scheme like this, though as long\r
+as notmuch can't split threads, I think it would be okay.\r
projects / notmuch-archives.git / commitdiff