Re: On disk tag storage format

diff --git a/67/1dacb3124b4b86f1f5ed559e9550c1155faf93 b/67/1dacb3124b4b86f1f5ed559e9550c1155faf93
new file mode 100644 (file)
index 0000000..483e4e9
--- /dev/null
+++ b/67/1dacb3124b4b86f1f5ed559e9550c1155faf93
@@ -0,0 +1,84 @@
+Return-Path: <tomi.ollila@iki.fi>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 983B4431FBC\r
+       for <notmuch@notmuchmail.org>; Thu, 29 Nov 2012 23:31:38 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id LJ4X4bczkd4O for <notmuch@notmuchmail.org>;\r
+       Thu, 29 Nov 2012 23:31:34 -0800 (PST)\r
+Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])\r
+       by olra.theworths.org (Postfix) with ESMTP id D99F1431FAF\r
+       for <notmuch@notmuchmail.org>; Thu, 29 Nov 2012 23:31:33 -0800 (PST)\r
+Received: from guru.guru-group.fi (localhost [IPv6:::1])\r
+       by guru.guru-group.fi (Postfix) with ESMTP id 6E1D91000E5;\r
+       Fri, 30 Nov 2012 09:31:31 +0200 (EET)\r
+From: Tomi Ollila <tomi.ollila@iki.fi>\r
+To: Eirik Byrkjeflot Anonsen <eirik@eirikba.org>,\r
+       notmuch mailing list <notmuch@notmuchmail.org>\r
+Subject: Re: On disk tag storage format\r
+In-Reply-To: <874nk8td7p.fsf@star.eba>\r
+References: <874nk8v9zw.fsf@zancas.localnet> <874nk8td7p.fsf@star.eba>\r
+User-Agent: Notmuch/0.14+116~g29fcdb5 (http://notmuchmail.org) Emacs/24.2.1\r
+       (x86_64-unknown-linux-gnu)\r
+X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL\r
+       $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F\r
+       !)g;OY^,BjTbr)Np:%c_o'jj,Z\r
+Date: Fri, 30 Nov 2012 09:31:31 +0200\r
+Message-ID: <m2d2yvlf70.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 30 Nov 2012 07:31:38 -0000\r
+\r
+On Thu, Nov 29 2012, Eirik Byrkjeflot Anonsen <eirik@eirikba.org> wrote:\r
+\r
+> David Bremner <david@tethera.net> writes:\r
+>\r
+>> Austin outlined on IRC a way of representing tags on disk as hardlinks\r
+>> to messages. In order to make the discussion more concrete, I wrote a\r
+>> prototype in python to dump the notmuch database to this format. On my\r
+>> 250k messages, this creates 40k new hardlinks, and uses about 5M of\r
+>> diskspace. The dump process takes about 20s on\r
+>> my core i7 machine.  With symbolic links, the same database takes about\r
+>> 150M of disk space; this isn't great but it isn't unbearable either.\r
+>\r
+> And eating 40k inodes, I suppose.  Which may matter to some systems.\r
+> (Hardlinks do not use extra inodes, as they are just directory entries\r
+> pointing to already existing inodes).\r
+>\r
+> Of course, the space usage also depends on the file system, as e.g. ext2\r
+> would use 1 complete block (typically 4kiB) to store the file name\r
+> pointed to per symlink.  ReiserFS would probably use 5M for the\r
+> directory entries and another 5M for the symlink data (wild guess).\r
+\r
+IIRC in mid 1990's (some) frisbee fs stored many symbolic links to one\r
+inode and, at the same time, stored multiple link names to same fs block\r
+... note that IIRC :D\r
+\r
+> eirik\r
+\r
+Tomi\r
+\r
+> _______________________________________________\r
+> notmuch mailing list\r
+> notmuch@notmuchmail.org\r
+> http://notmuchmail.org/mailman/listinfo/notmuch\r