MIME-Version: 1.0
In-Reply-To: <87mx21rw4h.fsf@servo.finestructure.net>
References:
 <CA+Tk8fwq2thNeKHgfG-EX0hgR7uyqrSce0ZMOhEJBsz1RVtRqg@mail.gmail.com>
	<20120811094635.GY28321@pub.cz.oracle.com>
	<CA+Tk8fzRp0DFfTnLaN3Biq_vEcBGzLg-=-af64Rb5DSM6yw25g@mail.gmail.com>
	<87mx21rw4h.fsf@servo.finestructure.net>
Date: Mon, 13 Aug 2012 22:10:23 +0300
Message-ID:
 <CA+Tk8fySXQN8Qv6QuLM-cJ4E+W9WBkws1B7RWa46C-_NKJeOGQ@mail.gmail.com>
Subject: Re: Alternative (raw) message store (i.e. instead of maildir)
From: Ciprian Dorin Craciun <ciprian.craciun@gmail.com>
To: Jameson Graef Rollins <jrollins@finestructure.net>
Content-Type: text/plain; charset=UTF-8
Cc: notmuch@notmuchmail.org
Precedence: list

On Sat, Aug 11, 2012 at 11:50 PM, Jameson Graef Rollins
<jrollins@finestructure.net> wrote:
> On Sat, Aug 11 2012, Ciprian Dorin Craciun <ciprian.craciun@gmail.com> wrote:
>>     My problem with it is that it doesn't scale... And I don't mean
>> this in a theoretical sense, I mean it in the concrete one: I have
>> about 661k emails... And a single `notmuch sync` takes a few tens of
>> seconds...
>
> Hey, Ciprian.  That sounds really slow, which makes me wonder if there
> are other things going on here.
> I have 155k messages, but notmuch new
> takes a fraction of a second for me.  This initial indexing certainly
> takes a long time (hours potentially), but additions after that should
> be really fast.  What version of notmuch are you using?  What version of
> xapian?


    Don't think there is anything wrong here... Its just drags with
the file system...

    So just to give a complete info:
    * hardware: Core i5, 8GiB RAM (7.5GiB of which is the FS cache),
SSD (about 175MiB raw disk access);
    * `notmuch --version`: 0.13 (built from sources on latest ArchLinux);
    * `notmuch count`: 701820;
    * `notmuch new` (after adding 5925 new emails, at touching others):
~~~~
Processed 7017 total files in 3m 19s (35 files/sec.).
Added 6061 new messages to the database. Detected 1116 file renames.
~~~~
    * actually the entire thing took almost 5 minutes, but the first
two it didn't display anything just acesing the disk;
    * `notmuch new` (another go, but this time I've `time`-d it):
~~~~
No new mail.
real    0m40.546s
user    0m4.523s
sys     0m17.506s
~~~~
    * `notmuch new` (yet another go, no change):
~~~~
No new mail.
real    0m39.190s
user    0m4.229s
sys     0m17.697s
~~~~
    * just to `du` the maildir (there are also 40k other files in
other maildirs not included in this count):
~~~~
8.7G    ......
real    0m22.229s
user    0m1.023s
sys     0m7.890s
~~~~
    * on `new` no hooks are run;
    * the file system in cause is JFS;


    As such I doubt the problem is with notmuch itself, and I guess
it's the file system interaction...

    Now I know I have a really obscure corner case, and I'm positively
amazed on how good notmuch handles this situation. I just wandered if
I could have fixed my problem by moving to an embedded DB, thus
skipping all that syscall overhead...

    Ciprian.