--- /dev/null
+Return-Path: <ciprian.craciun@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 23AE9431FC2\r
+ for <notmuch@notmuchmail.org>; Tue, 14 Aug 2012 09:38:26 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.799\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,\r
+ FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id JbwhBHq76UMo for <notmuch@notmuchmail.org>;\r
+ Tue, 14 Aug 2012 09:38:25 -0700 (PDT)\r
+Received: from mail-wg0-f45.google.com (mail-wg0-f45.google.com\r
+ [74.125.82.45]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client\r
+ certificate requested) by olra.theworths.org (Postfix) with ESMTPS id\r
+ 2310A431FAE for <notmuch@notmuchmail.org>; Tue, 14 Aug 2012 09:38:25 -0700\r
+ (PDT)\r
+Received: by wgbdq12 with SMTP id dq12so498176wgb.2\r
+ for <notmuch@notmuchmail.org>; Tue, 14 Aug 2012 09:38:22 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;\r
+ h=mime-version:in-reply-to:references:date:message-id:subject:from:to\r
+ :content-type; bh=pDHM2SnMCIian2BbvOSvTGSNB0/0/BQMahDhcEkTjQo=;\r
+ b=olQ4a6P8fw0+XRUM73Xr8heOeUKYVU/5mMVbr8sWJzJqjTyfDiymmMAr0dFw6u++fS\r
+ 422o/gREtlppqABZXWTsceTd2hEhTjHSNTjJbZCNX3ZneBnn2CiSbGgn45S9UXOM9rNj\r
+ mW17kpgQDqkO5TpJiVDHXhWk/7+WVNgC+I/Go/lS9fESXG5LiA3mehY7ZpqMrdbDHYSr\r
+ +5dfnm00NhZThdl8ysk/9D8P41DQswoOAfU8XMcCMpX+gsoIaWW2t9ewy+1FBKhII0al\r
+ NSReFTvXAxf8jaU+rUNVgGA+8i3tNDAdGTy7gMOcpvt1kqhd9XMbVblkPyXDpgSBk11W\r
+ UDiQ==\r
+MIME-Version: 1.0\r
+Received: by 10.180.94.164 with SMTP id dd4mr29323720wib.1.1344962302574; Tue,\r
+ 14 Aug 2012 09:38:22 -0700 (PDT)\r
+Received: by 10.180.104.196 with HTTP; Tue, 14 Aug 2012 09:38:22 -0700 (PDT)\r
+In-Reply-To: <20120814160442.GO28321@pub.cz.oracle.com>\r
+References:\r
+ <CA+Tk8fwq2thNeKHgfG-EX0hgR7uyqrSce0ZMOhEJBsz1RVtRqg@mail.gmail.com>\r
+ <20120811094635.GY28321@pub.cz.oracle.com> <874no613ms.fsf@flamingspork.com>\r
+ <20120814160442.GO28321@pub.cz.oracle.com>\r
+Date: Tue, 14 Aug 2012 19:38:22 +0300\r
+Message-ID:\r
+ <CA+Tk8fwVwWewTS-AVaaapQpLNU6a698acp-_ZmnktJ5ynRrx1A@mail.gmail.com>\r
+Subject: Re: Alternative (raw) message store (i.e. instead of maildir)\r
+From: Ciprian Dorin Craciun <ciprian.craciun@gmail.com>\r
+To: Stewart Smith <stewart@flamingspork.com>, notmuch@notmuchmail.org\r
+Content-Type: text/plain; charset=UTF-8\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 14 Aug 2012 16:38:26 -0000\r
+\r
+On Tue, Aug 14, 2012 at 7:04 PM, Vladimir Marek\r
+<Vladimir.Marek@oracle.com> wrote:\r
+>> > - fuse zip stores all changes in memory until unmounted\r
+>> > - fuse zip (and libzip for that matter) creates new temporary file when\r
+>> > updating archive, which takes considerable time when the archive is\r
+>> > very big.\r
+>>\r
+>> This isn't much of a hastle if you have maildir per time period and\r
+>> archive off. Maybe if you sync flags it may be...\r
+>\r
+> That might be interesting solution, maildir per time period.\r
+\r
+\r
+ Although using a zip file through FUSE as a maildir store is not\r
+much better in my opinion.\r
+\r
+ This is because it still doesn't solve the syscall overhead. For\r
+example just going through the list of files to find those that\r
+changed requires the following syscalls:\r
+ * reading the next directory entry (which is amortized as it reads\r
+them in a batch, but the batch size is limited, should we say 1\r
+syscall per 10 files?);\r
+ * stat-ing the file;\r
+\r
+ Now by adding FUSE we add an extra context switch for each syscall...\r
+\r
+ Although this issue would be problematic only for reindexing, but still...\r
+\r
+\r
+> But still\r
+> fuse zip caches all the data until unmounted. So even with just reading\r
+> it keeps growing (I hope I'm not accusing fuse zip here, but this is my\r
+> understanding form the code). This could be simply alleviated by having\r
+> it periodically unmounted and mounted again (perhaps from cron).\r
+\r
+ I think there is an option for FUSE mount to specify if the data\r
+should be cached by the kernel or not, as such this shouldn't be a\r
+problem for FUSE itself, except if the Zip FUSE handler does some\r
+extra caching.)\r
+\r
+\r
+>> > Of course this solution would have some disadvantages too, but for me\r
+>> > the advantages would win. At the moment I'm not sure if I want to\r
+>> > continue working on that. Maybe if there would be more interested guys\r
+>>\r
+>> I'm *really* tempted to investigate making this work for archived\r
+>> mail. Of course, the list of mounted file systems could get insane\r
+>> depending on granularity I guess...\r
+>\r
+> Well, if your granularity will be one archive per year of mail, it\r
+> should not be that bad ...\r
+\r
+\r
+ On the other hand I strongly sustain having a more optimized\r
+backend for emails, especially for such cases. For example a\r
+BerkeleyDB would perfectly fit such a use case, especially if we store\r
+the body and the headers in separate databases.\r
+\r
+ Just a small experiment, below are the R `summary(emails)` of the\r
+sizes of my 700k emails:\r
+~~~~\r
+ Min. 1st Qu. Median Mean 3rd Qu. Max.\r
+ 8 4364 5374 11510 7042 31090000\r
+~~~~\r
+\r
+ As seen 75% of the emails are below 7k, and this without any compression...\r
+\r
+ Moreover we could organize the keys so that in a B-Tree structure\r
+the emails in the same thread are closer together...\r
+\r
+ Ciprian.\r
projects / notmuch-archives.git / commitdiff