--- /dev/null
+Return-Path: <stewart@flamingspork.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 3A712431FBC\r
+ for <notmuch@notmuchmail.org>; Sun, 14 Feb 2010 16:29:24 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.721\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.721 tagged_above=-999 required=5\r
+ tests=[AWL=-0.722, BAYES_50=0.001] autolearn=ham\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id LdXSoGrFFfL8 for <notmuch@notmuchmail.org>;\r
+ Sun, 14 Feb 2010 16:29:23 -0800 (PST)\r
+Received: from kaylee.flamingspork.com (kaylee.flamingspork.com\r
+ [74.207.245.61])\r
+ by olra.theworths.org (Postfix) with ESMTP id 2C829431FAE\r
+ for <notmuch@notmuchmail.org>; Sun, 14 Feb 2010 16:29:23 -0800 (PST)\r
+Received: from willster (localhost [127.0.0.1])\r
+ by kaylee.flamingspork.com (Postfix) with ESMTPS id D2F746393\r
+ for <notmuch@notmuchmail.org>; Mon, 15 Feb 2010 00:26:23 +0000 (UTC)\r
+Received: from flamingspork.com (localhost.localdomain [127.0.0.1])\r
+ by willster (Postfix) with ESMTPS id C7DD51023D8F\r
+ for <notmuch@notmuchmail.org>; Mon, 15 Feb 2010 11:29:18 +1100 (EST)\r
+Date: Mon, 15 Feb 2010 11:29:14 +1100\r
+From: Stewart Smith <stewart@flamingspork.com>\r
+To: notmuch@notmuchmail.org\r
+Message-ID: <20100215002914.GA22402@flamingspork.com>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+User-Agent: Mutt/1.5.20 (2009-06-14)\r
+Subject: [notmuch] Mail in git\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 15 Feb 2010 00:29:24 -0000\r
+\r
+So... I sketched this out in my head at LCA... and it's taken a bit of\r
+time to actually properly try it.\r
+\r
+The problem is:\r
+A simple 'find ~/Maildir` takes 10 minutes, and if you write the\r
+output to a file, it's 88MB+\r
+\r
+there's "only" about 900,000 entries there. But this means 900,000\r
+files, which is a non-trivial amount. Some mail folders are quite\r
+large too.\r
+\r
+Some of this problem could just be solved by using notmuch a bit\r
+differently (folder per month for example).\r
+\r
+However... this is a one-way change and going back would be very\r
+tricky.\r
+\r
+There's also the backup problem. Iterating through ~1million inodes\r
+takes a *LONG* time. Restoring it takes even longer (think about\r
+writing all that data to the file system journal).\r
+\r
+Historically, if i'm running a backup, I couldn't really use my\r
+laptop, it'd be saturated with disk IO performing the file system\r
+dump. It would also take many hours.\r
+\r
+Restoring from backup? about 8hrs.\r
+\r
+An observation is that mail never changes. It may be reclassified (and\r
+that's what notmuch is for), but it never changes.\r
+\r
+We really just want a way to store and access many many many small\r
+blobs of data that never change.\r
+\r
+It turns out git is pretty good at that. Underneath, we could just use\r
+it as an object store (a simple git-hash-object and git-cat-file test\r
+confirmed this to be pretty simple to do). even better is since a lot\r
+of mail is fairly similar, to use delta compression between mail\r
+messages to reduce the storage space. Git is pretty good at that too.\r
+\r
+A few giant git packs will be much quicker to backup and restore than\r
+1million files.\r
+\r
+So... I wrote a script to test it....\r
+\r
+$ time perl /home/stewart/evenless.pl /home/stewart/Maildir/\r
+\r
+real 841m41.491s\r
+user 491m3.200s\r
+sys 261m58.080s\r
+\r
+Which goes from a 15GB Maildir to a 3.7GB git repo.\r
+\r
+The algorithm of evenless.pl is basically:\r
+1 get next directory entry\r
+2 if is directory, recurse into it\r
+3 write item to git (git hash-object -w)\r
+4 add item to tree object\r
+5 if number of items written = 1000\r
+ 5.1 make pack of last 1000 items\r
+6 goto 1\r
+\r
+$ git count-objects -v\r
+count: 479\r
+size: 27680\r
+in-pack: 873109\r
+packs: 1084\r
+size-pack: 3746219\r
+prune-packable: 0\r
+garbage: 0\r
+\r
+If i did a "git checkout", about 8 hours later i'd have a directory\r
+tree exactly the same as my maildir.\r
+\r
+Why didn't I just git-add everything? I didn't exactly feel like\r
+creating another giant copy of my mail (that also takes a long time).\r
+\r
+What about adding more mail to the archive?\r
+\r
+So the way I think is that you use a Maildir for day to day mail (e.g.\r
+delivery) and every so often you run some magic command that takes old\r
+mail out of the Maildir and stores it in the git repo.\r
+\r
+Next step?\r
+\r
+Make notmuch be able to read mail out of it and add it to an index\r
+(oh, and some kind of verification and error checking about creating\r
+the git repo).\r
+-- \r
+Stewart Smith\r
projects / notmuch-archives.git / commitdiff