--- /dev/null
+Return-Path: <Vladimir.Marek@Oracle.COM>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id BE42C431FC2\r
+ for <notmuch@notmuchmail.org>; Tue, 14 Aug 2012 09:52:08 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -4.999\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-4.999 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_HI=-5, UNPARSEABLE_RELAY=0.001]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 0veuzWsXCWt6 for <notmuch@notmuchmail.org>;\r
+ Tue, 14 Aug 2012 09:52:08 -0700 (PDT)\r
+Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 2F756431FAE\r
+ for <notmuch@notmuchmail.org>; Tue, 14 Aug 2012 09:52:08 -0700 (PDT)\r
+Received: from acsinet22.oracle.com (acsinet22.oracle.com [141.146.126.238])\r
+ by rcsinet15.oracle.com (Sentrion-MTA-4.2.2/Sentrion-MTA-4.2.2) with\r
+ ESMTP id q7EGq4T4021055\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);\r
+ Tue, 14 Aug 2012 16:52:05 GMT\r
+Received: from acsmt357.oracle.com (acsmt357.oracle.com [141.146.40.157])\r
+ by acsinet22.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id\r
+ q7EGq4YB004735\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);\r
+ Tue, 14 Aug 2012 16:52:04 GMT\r
+Received: from abhmt104.oracle.com (abhmt104.oracle.com [141.146.116.56])\r
+ by acsmt357.oracle.com (8.12.11.20060308/8.12.11) with ESMTP id\r
+ q7EGq3jt001878; Tue, 14 Aug 2012 11:52:03 -0500\r
+Received: from pub.cz.oracle.com (/10.163.20.32)\r
+ by default (Oracle Beehive Gateway v4.0)\r
+ with ESMTP ; Tue, 14 Aug 2012 09:52:03 -0700\r
+Date: Tue, 14 Aug 2012 18:50:44 +0200\r
+From: Vladimir Marek <Vladimir.Marek@Oracle.COM>\r
+To: Ciprian Dorin Craciun <ciprian.craciun@gmail.com>\r
+Subject: Re: Alternative (raw) message store (i.e. instead of maildir)\r
+Message-ID: <20120814165044.GP28321@pub.cz.oracle.com>\r
+Mail-Followup-To: Ciprian Dorin Craciun <ciprian.craciun@gmail.com>,\r
+ Stewart Smith <stewart@flamingspork.com>, notmuch@notmuchmail.org\r
+References:\r
+ <CA+Tk8fwq2thNeKHgfG-EX0hgR7uyqrSce0ZMOhEJBsz1RVtRqg@mail.gmail.com>\r
+ <20120811094635.GY28321@pub.cz.oracle.com> <874no613ms.fsf@flamingspork.com>\r
+ <20120814160442.GO28321@pub.cz.oracle.com>\r
+ <CA+Tk8fwVwWewTS-AVaaapQpLNU6a698acp-_ZmnktJ5ynRrx1A@mail.gmail.com>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Disposition: inline\r
+In-Reply-To:\r
+ <CA+Tk8fwVwWewTS-AVaaapQpLNU6a698acp-_ZmnktJ5ynRrx1A@mail.gmail.com>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Source-IP: acsinet22.oracle.com [141.146.126.238]\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 14 Aug 2012 16:52:08 -0000\r
+\r
+> >> > - fuse zip stores all changes in memory until unmounted\r
+> >> > - fuse zip (and libzip for that matter) creates new temporary file when\r
+> >> > updating archive, which takes considerable time when the archive is\r
+> >> > very big.\r
+> >>\r
+> >> This isn't much of a hastle if you have maildir per time period and\r
+> >> archive off. Maybe if you sync flags it may be...\r
+> >\r
+> > That might be interesting solution, maildir per time period.\r
+> \r
+> \r
+> Although using a zip file through FUSE as a maildir store is not\r
+> much better in my opinion.\r
+> \r
+> This is because it still doesn't solve the syscall overhead. For\r
+> example just going through the list of files to find those that\r
+> changed requires the following syscalls:\r
+> * reading the next directory entry (which is amortized as it reads\r
+> them in a batch, but the batch size is limited, should we say 1\r
+> syscall per 10 files?);\r
+> * stat-ing the file;\r
+> \r
+> Now by adding FUSE we add an extra context switch for each syscall...\r
+> \r
+> Although this issue would be problematic only for reindexing, but still...\r
+\r
+That's a price I would be willing to pay to have single file instead of\r
+many.\r
+\r
+\r
+\r
+\r
+> > But still\r
+> > fuse zip caches all the data until unmounted. So even with just reading\r
+> > it keeps growing (I hope I'm not accusing fuse zip here, but this is my\r
+> > understanding form the code). This could be simply alleviated by having\r
+> > it periodically unmounted and mounted again (perhaps from cron).\r
+> \r
+> I think there is an option for FUSE mount to specify if the data\r
+> should be cached by the kernel or not, as such this shouldn't be a\r
+> problem for FUSE itself, except if the Zip FUSE handler does some\r
+> extra caching.)\r
+\r
+To my understanding it's the handler itself.\r
+\r
+\r
+\r
+\r
+> >> > Of course this solution would have some disadvantages too, but for me\r
+> >> > the advantages would win. At the moment I'm not sure if I want to\r
+> >> > continue working on that. Maybe if there would be more interested guys\r
+> >>\r
+> >> I'm *really* tempted to investigate making this work for archived\r
+> >> mail. Of course, the list of mounted file systems could get insane\r
+> >> depending on granularity I guess...\r
+> >\r
+> > Well, if your granularity will be one archive per year of mail, it\r
+> > should not be that bad ...\r
+> \r
+> \r
+> On the other hand I strongly sustain having a more optimized\r
+> backend for emails, especially for such cases. For example a\r
+> BerkeleyDB would perfectly fit such a use case, especially if we store\r
+> the body and the headers in separate databases.\r
+> \r
+> Just a small experiment, below are the R `summary(emails)` of the\r
+> sizes of my 700k emails:\r
+> ~~~~\r
+> Min. 1st Qu. Median Mean 3rd Qu. Max.\r
+> 8 4364 5374 11510 7042 31090000\r
+> ~~~~\r
+> \r
+> As seen 75% of the emails are below 7k, and this without any compression...\r
+> \r
+> Moreover we could organize the keys so that in a B-Tree structure\r
+> the emails in the same thread are closer together...\r
+\r
+Now I'm not sure if you talk about some berkeley-db fuse filesystem or\r
+direct support in notmuch. I don't have enough cycles to modify notmuch,\r
+so I started to look at simpler (codewise) solution ...\r
+\r
+To summarize, what I personally want from the mail storage\r
+\r
+- ability to read and write mails\r
+- should work with mutt (or mutt-kz)\r
+- simple backup to windows drive (files can't contain double colon ':')\r
+\r
+-- \r
+ Vlad\r
projects / notmuch-archives.git / commitdiff