Re: On disk tag storage format

diff --git a/cc/4eb726b18604a45a63b033d60f78682aaebcca b/cc/4eb726b18604a45a63b033d60f78682aaebcca
new file mode 100644 (file)
index 0000000..b59a750
--- /dev/null
+++ b/cc/4eb726b18604a45a63b033d60f78682aaebcca
@@ -0,0 +1,74 @@
+Return-Path: <eirik@eirikba.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 7050C431FAF\r
+       for <notmuch@notmuchmail.org>; Thu, 29 Nov 2012 11:34:58 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id gweATqeNm2zi for <notmuch@notmuchmail.org>;\r
+       Thu, 29 Nov 2012 11:34:54 -0800 (PST)\r
+Received: from atmail.labs2.com (atmail.labs2.com [93.182.166.49])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 5A570431FAE\r
+       for <notmuch@notmuchmail.org>; Thu, 29 Nov 2012 11:34:54 -0800 (PST)\r
+Received: from [178.74.1.248] (helo=star.eba)\r
+       by atmail.labs2.com with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77)\r
+       (envelope-from <eirik@eirikba.org>) id 1Te9sh-0006gE-Hf\r
+       for notmuch@notmuchmail.org; Thu, 29 Nov 2012 20:34:43 +0100\r
+Received: from eirik by star.eba with local (Exim 4.80)\r
+       (envelope-from <eirik@eirikba.org>) id 1Te9so-0002WP-70\r
+       for notmuch@notmuchmail.org; Thu, 29 Nov 2012 20:34:50 +0100\r
+From: Eirik Byrkjeflot Anonsen <eirik@eirikba.org>\r
+To: "notmuch mailing list" <notmuch@notmuchmail.org>\r
+Subject: Re: On disk tag storage format\r
+References: <874nk8v9zw.fsf@zancas.localnet>\r
+Date: Thu, 29 Nov 2012 20:34:50 +0100\r
+In-Reply-To: <874nk8v9zw.fsf@zancas.localnet> (David Bremner's message of\r
+       "Thu, 29 Nov 2012 09:01:23 -0400")\r
+Message-ID: <874nk8td7p.fsf@star.eba>\r
+User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux)\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+X-ACL-Warn: Authenticated as:  Sent as: eirik@eirikba.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 29 Nov 2012 19:34:58 -0000\r
+\r
+David Bremner <david@tethera.net> writes:\r
+\r
+> Austin outlined on IRC a way of representing tags on disk as hardlinks\r
+> to messages. In order to make the discussion more concrete, I wrote a\r
+> prototype in python to dump the notmuch database to this format. On my\r
+> 250k messages, this creates 40k new hardlinks, and uses about 5M of\r
+> diskspace. The dump process takes about 20s on\r
+> my core i7 machine.  With symbolic links, the same database takes about\r
+> 150M of disk space; this isn't great but it isn't unbearable either.\r
+\r
+And eating 40k inodes, I suppose.  Which may matter to some systems.\r
+(Hardlinks do not use extra inodes, as they are just directory entries\r
+pointing to already existing inodes).\r
+\r
+Of course, the space usage also depends on the file system, as e.g. ext2\r
+would use 1 complete block (typically 4kiB) to store the file name\r
+pointed to per symlink.  ReiserFS would probably use 5M for the\r
+directory entries and another 5M for the symlink data (wild guess).\r
+\r
+eirik\r