From: Eirik Byrkjeflot Anonsen Date: Thu, 29 Nov 2012 19:34:50 +0000 (+0100) Subject: Re: On disk tag storage format X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=d390a154781f287f2cbe9b1ed50431758f5773c6;p=notmuch-archives.git Re: On disk tag storage format --- diff --git a/cc/4eb726b18604a45a63b033d60f78682aaebcca b/cc/4eb726b18604a45a63b033d60f78682aaebcca new file mode 100644 index 000000000..b59a75060 --- /dev/null +++ b/cc/4eb726b18604a45a63b033d60f78682aaebcca @@ -0,0 +1,74 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 7050C431FAF + for ; Thu, 29 Nov 2012 11:34:58 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id gweATqeNm2zi for ; + Thu, 29 Nov 2012 11:34:54 -0800 (PST) +Received: from atmail.labs2.com (atmail.labs2.com [93.182.166.49]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 5A570431FAE + for ; Thu, 29 Nov 2012 11:34:54 -0800 (PST) +Received: from [178.74.1.248] (helo=star.eba) + by atmail.labs2.com with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) + (envelope-from ) id 1Te9sh-0006gE-Hf + for notmuch@notmuchmail.org; Thu, 29 Nov 2012 20:34:43 +0100 +Received: from eirik by star.eba with local (Exim 4.80) + (envelope-from ) id 1Te9so-0002WP-70 + for notmuch@notmuchmail.org; Thu, 29 Nov 2012 20:34:50 +0100 +From: Eirik Byrkjeflot Anonsen +To: "notmuch mailing list" +Subject: Re: On disk tag storage format +References: <874nk8v9zw.fsf@zancas.localnet> +Date: Thu, 29 Nov 2012 20:34:50 +0100 +In-Reply-To: <874nk8v9zw.fsf@zancas.localnet> (David Bremner's message of + "Thu, 29 Nov 2012 09:01:23 -0400") +Message-ID: <874nk8td7p.fsf@star.eba> +User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux) +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-ACL-Warn: Authenticated as: Sent as: eirik@eirikba.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 29 Nov 2012 19:34:58 -0000 + +David Bremner writes: + +> Austin outlined on IRC a way of representing tags on disk as hardlinks +> to messages. In order to make the discussion more concrete, I wrote a +> prototype in python to dump the notmuch database to this format. On my +> 250k messages, this creates 40k new hardlinks, and uses about 5M of +> diskspace. The dump process takes about 20s on +> my core i7 machine. With symbolic links, the same database takes about +> 150M of disk space; this isn't great but it isn't unbearable either. + +And eating 40k inodes, I suppose. Which may matter to some systems. +(Hardlinks do not use extra inodes, as they are just directory entries +pointing to already existing inodes). + +Of course, the space usage also depends on the file system, as e.g. ext2 +would use 1 complete block (typically 4kiB) to store the file name +pointed to per symlink. ReiserFS would probably use 5M for the +directory entries and another 5M for the symlink data (wild guess). + +eirik