--- /dev/null
+Return-Path: <amdragon@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id B9862431FB6\r
+ for <notmuch@notmuchmail.org>; Wed, 4 May 2011 18:48:41 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.699\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.699 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001,\r
+ RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id TlrG3flThR9F for <notmuch@notmuchmail.org>;\r
+ Wed, 4 May 2011 18:48:40 -0700 (PDT)\r
+Received: from mail-qy0-f181.google.com (mail-qy0-f181.google.com\r
+ [209.85.216.181]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id B6023431FB5\r
+ for <notmuch@notmuchmail.org>; Wed, 4 May 2011 18:48:40 -0700 (PDT)\r
+Received: by qyg14 with SMTP id 14so1431099qyg.5\r
+ for <notmuch@notmuchmail.org>; Wed, 04 May 2011 18:48:40 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;\r
+ h=domainkey-signature:mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+ :content-transfer-encoding;\r
+ bh=BO7LfIVZLYpcGxAj9Z+5N3lBRVQpfg/L5yE36ODa2xk=;\r
+ b=mGFEUE5sjtmC7HhgilOwy5j+Eph6ZQ/FbYrQm+/OUR8XXRMn5SvrwMJbOXcXFncebG\r
+ Q+VRz+Ta1HF+tYxT1AlfQuU8Igdb6tPgh67gt7Bj7p6WUbQRof4K5ui76gJSywvMVUD3\r
+ HN8BdwfRHy0LQn6Rie+oLFr5tlsOXVvz30tLU=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;\r
+ h=mime-version:sender:in-reply-to:references:date\r
+ :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+ :content-transfer-encoding;\r
+ b=Xgg6mmrH0SsDq1yxMI4QyhiSijn+J0dyRVqZrsac6o07cQhzZlH2g5Egn5QOlUa68O\r
+ Mcg23ZCRVx3b2aOLyvV5LqRr9WaFp+fGxcTTpPGhSxrcVtvr7MGRQ45p7ZtULHoB+zeV\r
+ HFTDW1oQ/LKqBT1i5Fa8t5kFK6hW+FPdmeBP0=\r
+MIME-Version: 1.0\r
+Received: by 10.229.206.42 with SMTP id fs42mr1405286qcb.150.1304560119908;\r
+ Wed, 04 May 2011 18:48:39 -0700 (PDT)\r
+Sender: amdragon@gmail.com\r
+Received: by 10.229.233.17 with HTTP; Wed, 4 May 2011 18:48:39 -0700 (PDT)\r
+In-Reply-To: <m3sjsv2kw2.fsf@zsu.kismala.com>\r
+References: <m3sjsv2kw2.fsf@zsu.kismala.com>\r
+Date: Wed, 4 May 2011 21:48:39 -0400\r
+X-Google-Sender-Auth: SOnPQT98FKqvGwp6EFEUpJTihRI\r
+Message-ID: <BANLkTimvs8ttQCL0AK9UrjcxxgaeqQ9fSg@mail.gmail.com>\r
+Subject: Re: storing From and Subject in xapian\r
+From: Austin Clements <amdragon@mit.edu>\r
+To: Istvan Marko <notmuch@kismala.com>\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+Content-Transfer-Encoding: quoted-printable\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 05 May 2011 01:48:41 -0000\r
+\r
+This is awesome. What was your machine configuration?\r
+\r
+As another data point, with a probably very different configuration (8\r
+year old P4, new SSD), my test query was 1.9X faster uncached and 1.6X\r
+faster cached. It also produced 60% fewer disk reads. I saw the same\r
+1% increase in database size.\r
+\r
+BTW, the reason you're missing some of the subjects is that the char*\r
+returned from _notmuch_message_get_header_value goes out of scope as\r
+soon as that function returns. A simple fix is to replace\r
+ return value.c_str();\r
+with\r
+ return talloc_strdup (message, value.c_str ());\r
+\r
+Values are probably the right place to store this information (though\r
+I've never been completely clear on the difference between document\r
+data and values). Terms would be indexed, which is both unnecessary\r
+(unless there's a reason to do *exact* matches on from and subject?)\r
+and would result in more database expansion.\r
+\r
+On Tue, May 3, 2011 at 11:40 PM, Istvan Marko <notmuch@kismala.com> wrote:\r
+>\r
+> I have been looking at the I/O patterns of "notmuch search" with the\r
+> default output format and noticed that it has to parse the maildir file\r
+> of every matched message to get the From and Subject headers. I figured\r
+> that this must be slowing things down, especially when the files are not\r
+> in the filesystem cache.\r
+>\r
+> So I wanted to see how much difference would it make to have the From\r
+> and Subject stored in xapian to avoid this parsing.\r
+>\r
+> With the attached patch I get a speedup of 2x with cached and almost 10x\r
+> with uncached files for searches with many matches.\r
+>\r
+> The attached patch is only intended as proof of concept. I am not\r
+> familiar with xapian so I wasn't sure if this kind of data should be\r
+> stored as terms, values or data. I went with values simply because I saw\r
+> that message-id and timestamp were already stored that way. Perhaps the\r
+> data type would be more appropriate since the fields are not used for\r
+> searching or sorting. Oh and for some reason I get blank Subject for\r
+> about 1% of the matches.\r
+>\r
+>\r
+> Is there a downside to this approach? The only one I see is that the\r
+> xapian db size increases by about 1% but to me the speed increase would\r
+> be well worth it.\r
+>\r
+>\r
+>\r
+>\r
+> --\r
+> =A0 =A0 =A0 =A0Istvan\r
+>\r
+> _______________________________________________\r
+> notmuch mailing list\r
+> notmuch@notmuchmail.org\r
+> http://notmuchmail.org/mailman/listinfo/notmuch\r
+>\r
+>\r
projects / notmuch-archives.git / commitdiff