From e16a284474087d0693dba579cdcd38347637dea8 Mon Sep 17 00:00:00 2001 From: Ciprian Dorin Craciun Date: Mon, 13 Aug 2012 22:10:23 +0300 Subject: [PATCH] Re: Alternative (raw) message store (i.e. instead of maildir) --- 8c/1fbb263f24862ee0ecf1b492c02ef88a1ca8fc | 133 ++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 8c/1fbb263f24862ee0ecf1b492c02ef88a1ca8fc diff --git a/8c/1fbb263f24862ee0ecf1b492c02ef88a1ca8fc b/8c/1fbb263f24862ee0ecf1b492c02ef88a1ca8fc new file mode 100644 index 000000000..8ec15e07d --- /dev/null +++ b/8c/1fbb263f24862ee0ecf1b492c02ef88a1ca8fc @@ -0,0 +1,133 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id EC31A431FB6 + for ; Mon, 13 Aug 2012 12:10:27 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.799 +X-Spam-Level: +X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id NcIPQxOKR3Ca for ; + Mon, 13 Aug 2012 12:10:27 -0700 (PDT) +Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com + [209.85.212.173]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 28777431FAE + for ; Mon, 13 Aug 2012 12:10:27 -0700 (PDT) +Received: by wibhm6 with SMTP id hm6so3111131wib.2 + for ; Mon, 13 Aug 2012 12:10:24 -0700 (PDT) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; + h=mime-version:in-reply-to:references:date:message-id:subject:from:to + :cc:content-type; + bh=SSygKeDfqbu75WG6bTpEw2gqCIcpqdIMP+Tog7Pdric=; + b=vOEUmf1SKypFFdW6tbJnLLoI5qQaHxSuPZ9Id4fuFr9C5G5t6osPOuUoEd+TRi773z + 9EeuEmyiXIJCFi7Q2ucUu3BTFL/5kRpfJfKFHf5Kimu/tnJ751EKwX3SbTLj8IIown0f + n01hmVV+F6e0ffweBJ5lSaRSQu75hiz4Pn5+fhKv4XG348Ei3qViwG5S9RB+Lojtgify + rhV68V33/WZEXsgVxwe+R77IO1eqTDUjcmrFBgrkegYhquA2WL3butJjhu5mJXfjZ698 + xhbNpEhjRbKZ9/rGWneSSBAaCiVlLWGkQ6+5VafxjaIDmrjcgFr+p05ZJDZ9gFqVbk0J + iuQw== +MIME-Version: 1.0 +Received: by 10.216.195.143 with SMTP id p15mr6906724wen.167.1344885024096; + Mon, 13 Aug 2012 12:10:24 -0700 (PDT) +Received: by 10.180.104.196 with HTTP; Mon, 13 Aug 2012 12:10:23 -0700 (PDT) +In-Reply-To: <87mx21rw4h.fsf@servo.finestructure.net> +References: + + <20120811094635.GY28321@pub.cz.oracle.com> + + <87mx21rw4h.fsf@servo.finestructure.net> +Date: Mon, 13 Aug 2012 22:10:23 +0300 +Message-ID: + +Subject: Re: Alternative (raw) message store (i.e. instead of maildir) +From: Ciprian Dorin Craciun +To: Jameson Graef Rollins +Content-Type: text/plain; charset=UTF-8 +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 13 Aug 2012 19:10:28 -0000 + +On Sat, Aug 11, 2012 at 11:50 PM, Jameson Graef Rollins + wrote: +> On Sat, Aug 11 2012, Ciprian Dorin Craciun wrote: +>> My problem with it is that it doesn't scale... And I don't mean +>> this in a theoretical sense, I mean it in the concrete one: I have +>> about 661k emails... And a single `notmuch sync` takes a few tens of +>> seconds... +> +> Hey, Ciprian. That sounds really slow, which makes me wonder if there +> are other things going on here. +> I have 155k messages, but notmuch new +> takes a fraction of a second for me. This initial indexing certainly +> takes a long time (hours potentially), but additions after that should +> be really fast. What version of notmuch are you using? What version of +> xapian? + + + Don't think there is anything wrong here... Its just drags with +the file system... + + So just to give a complete info: + * hardware: Core i5, 8GiB RAM (7.5GiB of which is the FS cache), +SSD (about 175MiB raw disk access); + * `notmuch --version`: 0.13 (built from sources on latest ArchLinux); + * `notmuch count`: 701820; + * `notmuch new` (after adding 5925 new emails, at touching others): +~~~~ +Processed 7017 total files in 3m 19s (35 files/sec.). +Added 6061 new messages to the database. Detected 1116 file renames. +~~~~ + * actually the entire thing took almost 5 minutes, but the first +two it didn't display anything just acesing the disk; + * `notmuch new` (another go, but this time I've `time`-d it): +~~~~ +No new mail. +real 0m40.546s +user 0m4.523s +sys 0m17.506s +~~~~ + * `notmuch new` (yet another go, no change): +~~~~ +No new mail. +real 0m39.190s +user 0m4.229s +sys 0m17.697s +~~~~ + * just to `du` the maildir (there are also 40k other files in +other maildirs not included in this count): +~~~~ +8.7G ...... +real 0m22.229s +user 0m1.023s +sys 0m7.890s +~~~~ + * on `new` no hooks are run; + * the file system in cause is JFS; + + + As such I doubt the problem is with notmuch itself, and I guess +it's the file system interaction... + + Now I know I have a really obscure corner case, and I'm positively +amazed on how good notmuch handles this situation. I just wandered if +I could have fixed my problem by moving to an embedded DB, thus +skipping all that syscall overhead... + + Ciprian. -- 2.26.2