Re: Slowness (search opens every email file?)

diff --git a/da/ad5b1e468f76edd3ca1ca0f2379f875f4c865a b/da/ad5b1e468f76edd3ca1ca0f2379f875f4c865a
new file mode 100644 (file)
index 0000000..8e07b9f
--- /dev/null
+++ b/da/ad5b1e468f76edd3ca1ca0f2379f875f4c865a
@@ -0,0 +1,124 @@
+Return-Path: <patricktotzke@googlemail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 67358431FD0\r
+       for <notmuch@notmuchmail.org>; Mon, 11 Jul 2011 14:59:06 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.799\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5\r
+       tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,\r
+       FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id uzodx91vpLXP for <notmuch@notmuchmail.org>;\r
+       Mon, 11 Jul 2011 14:59:05 -0700 (PDT)\r
+Received: from mail-wy0-f181.google.com (mail-wy0-f181.google.com\r
+       [74.125.82.181]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 45EB9431FB6\r
+       for <notmuch@notmuchmail.org>; Mon, 11 Jul 2011 14:59:05 -0700 (PDT)\r
+Received: by wyh22 with SMTP id 22so3246825wyh.26\r
+       for <notmuch@notmuchmail.org>; Mon, 11 Jul 2011 14:59:04 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+       d=googlemail.com; s=gamma;\r
+       h=date:from:to:cc:subject:message-id:references:mime-version\r
+       :content-type:content-disposition:in-reply-to:user-agent;\r
+       bh=TBu5anTZ281vVZYZ7L6WbhuPO7dXP2BaSf5eE/0xBH4=;\r
+       b=VPrdNOIRwSioQiCZNIJuO86WxJPxizUWfNr/TC61/MMsLzsck5t169qDMqL+20LHab\r
+       ECFbVICu5a+WtFojEvF2jUfHmvFwys7fy8d4HAq74RHt+8eRbmZcGHAi3N/y/uRSeLQ2\r
+       epsr3NO0pC1JcdnticBUanT8rijBJlDLO2UJI=\r
+Received: by 10.216.235.157 with SMTP id u29mr3352438weq.24.1310421543803;\r
+       Mon, 11 Jul 2011 14:59:03 -0700 (PDT)\r
+Received: from brick.lan (cpc1-sgyl2-0-0-cust47.sgyl.cable.virginmedia.com\r
+       [80.192.18.48])\r
+       by mx.google.com with ESMTPS id fe4sm10365228wbb.11.2011.07.11.14.59.00\r
+       (version=TLSv1/SSLv3 cipher=OTHER);\r
+       Mon, 11 Jul 2011 14:59:02 -0700 (PDT)\r
+Date: Mon, 11 Jul 2011 22:58:58 +0100\r
+From: Patrick Totzke <patricktotzke@googlemail.com>\r
+To: Jason Woofenden <jason@jasonwoof.com>\r
+Subject: Re: Slowness (search opens every email file?)\r
+Message-ID: <20110711215858.GA18884@brick.lan>\r
+References: <20110711190721.GA5386@compy.jasonwoof.org>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; micalg=pgp-sha1;\r
+       protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5"\r
+Content-Disposition: inline\r
+In-Reply-To: <20110711190721.GA5386@compy.jasonwoof.org>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+Cc: Notmuch Mail <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 11 Jul 2011 21:59:06 -0000\r
+\r
+\r
+--bg08WKrSYDhXBjb5\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+Hi Jason,\r
+On Mon, Jul 11, 2011 at 03:07:21PM -0400, Jason Woofenden wrote:\r
+> notmuch search tag:foo is slow!\r
+>=20\r
+yes, i've just used the vim ui for the first time and i agreee, its sluggis=\r
+h,\r
+searching for * takes a while.\r
+\r
+> (when my e-mail files are not already in the disk cache)\r
+>=20\r
+> I saw on my activity monitor applet that it was using mostly i/o,\r
+> and started to wonder if it was opening every e-mail. I little work\r
+> with strace and searching revealed that this command was opening\r
+> many many e-mail files from my maildir(s). I spent a little while\r
+> digging around in the notmuch source, and didn't see where it was\r
+> opening the email files.\r
+\r
+I cannot reproduce this. I'm no expert, but at least the output of=20\r
+ strace vim -c ":NotMuch" 2>log\r
+does not contain any path that matches that of my maildir.\r
+Also, I wout be surprised if all individual mails would be read,\r
+because for this, a search for all messages feels too fast.\r
+\r
+> 2) Do you agree that it should instead get this information from\r
+> the database?\r
+agreed. And If the mail files get read on every search for you,\r
+something is definately going wrong there.\r
+\r
+A quick browse through notmuch.vim tells me that=20\r
+1) it doesn't use notmuchs json output, i think it should, as iirc\r
+this api is considered 'more stable' and is easier to parse than the=20\r
+default output. More importantly,\r
+2) the output of notmuch is copied into a list. This will of course\r
+be slow if your query matches a lot of messages. Could this be done\r
+by asyncronously writing to the buffer somehow?\r
+\r
+best,\r
+/p\r
+\r
+--bg08WKrSYDhXBjb5\r
+Content-Type: application/pgp-signature; name="signature.asc"\r
+Content-Description: Digital signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.11 (GNU/Linux)\r
+\r
+iEYEARECAAYFAk4bciIACgkQlDQDZ9fWxaon3wCgvstp9RjrttNwVaoKWGu+4yvs\r
+YTQAn0byopDOt2qtL68Xmvr1OxC20FpY\r
+=glvO\r
+-----END PGP SIGNATURE-----\r
+\r
+--bg08WKrSYDhXBjb5--\r