[notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

diff --git a/8c/26063f7a393c2c8756face4ba115b448fbac98 b/8c/26063f7a393c2c8756face4ba115b448fbac98
new file mode 100644 (file)
index 0000000..67db239
--- /dev/null
+++ b/8c/26063f7a393c2c8756face4ba115b448fbac98
@@ -0,0 +1,161 @@
+Return-Path: <madduck@lapse.rw.madduck.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id A0753431FBC\r
+       for <notmuch@notmuchmail.org>; Sun, 24 Jan 2010 16:47:35 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.599\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5\r
+       tests=[BAYES_00=-2.599] autolearn=ham\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id dd1WZkSwKsS2 for <notmuch@notmuchmail.org>;\r
+       Sun, 24 Jan 2010 16:47:34 -0800 (PST)\r
+Received: from clegg.madduck.net (clegg.madduck.net [193.242.105.96])\r
+       by olra.theworths.org (Postfix) with ESMTP id 4569C431FAE\r
+       for <notmuch@notmuchmail.org>; Sun, 24 Jan 2010 16:47:34 -0800 (PST)\r
+Received: from lapse.rw.madduck.net (leibniz.catalyst.net.nz [202.78.240.7])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (Client CN "lapse.rw.madduck.net",\r
+       Issuer "CAcert Class 3 Root" (verified OK))\r
+       by clegg.madduck.net (postfix) with ESMTPS id B8B901D4099;\r
+       Mon, 25 Jan 2010 01:47:23 +0100 (CET)\r
+Received: by lapse.rw.madduck.net (Postfix, from userid 1000)\r
+       id 5FC416C5; Mon, 25 Jan 2010 13:46:59 +1300 (NZDT)\r
+Date: Mon, 25 Jan 2010 13:46:59 +1300\r
+From: martin f krafft <madduck@madduck.net>\r
+To: Asheesh Laroia <asheesh@asheesh.org>\r
+Message-ID: <20100125004659.GA24818@lapse.rw.madduck.net>\r
+Mail-Followup-To: Asheesh Laroia <asheesh@asheesh.org>,\r
+       notmuch <notmuch@notmuchmail.org>\r
+References: <20100111221909.GA30299@lapse.rw.madduck.net>\r
+       <1263267603-sup-302@elise>\r
+       <20100112045152.GA15275@lapse.rw.madduck.net>\r
+       <alpine.DEB.2.00.1001140254240.27198@vellum>\r
+       <20100114203730.GE4691@lapse.rw.madduck.net>\r
+       <alpine.DEB.2.00.1001210124590.24778@localhost>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; micalg=pgp-ripemd160;\r
+       protocol="application/pgp-signature"; boundary="fdj2RfSjLxBAspz7"\r
+Content-Disposition: inline\r
+In-Reply-To: <alpine.DEB.2.00.1001210124590.24778@localhost>\r
+X-Motto: Keep the good times rollin'\r
+X-OS: Debian GNU/Linux squeeze/sid kernel 2.6.32-trunk-686 i686\r
+X-Spamtrap: madduck.bogus@madduck.net\r
+X-Subliminal-Message: debian/rules!\r
+User-Agent: Mutt/1.5.20 (2009-06-14)\r
+X-Virus-Scanned: clamav-milter 0.95.3 at clegg\r
+X-Virus-Status: Clean\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+Subject: [notmuch] Git as notmuch object store (was: Potential problem using\r
+ Git for mail)\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 25 Jan 2010 00:47:35 -0000\r
+\r
+\r
+--fdj2RfSjLxBAspz7\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+also sprach Asheesh Laroia <asheesh@asheesh.org> [2010.01.21.1928 +1300]:\r
+> >I suppose that I never actually considered merges on the IMAP\r
+> >server side, but obviously the IMAP server has to work off a clone,\r
+> >and that means it needs to merge.\r
+>=20\r
+> It's not "merge" that's unsafe; that just builds a tree in the git\r
+> index (assuming no conflicts). It's the ensuing process of git\r
+> writing a tree to the filesystem that is problematic.\r
+\r
+There is no way to make that atomic, I am afraid. As you say.\r
+\r
+> I could probably actually write a wrapper that locks the Maildir\r
+> while git is operating. It would probably be specific to each IMAP\r
+> server.\r
+\r
+Ouch! I'd really rather not go there.\r
+\r
+> Note that this mean git is fundamentally incompatible with\r
+> Maildir, not just IMAP servers.\r
+\r
+We had an idea about using Git to replace IMAP altogether, along\r
+with making notmuch use a bare Git repository as object store. The\r
+idea is that notmuch uses low-level Git commands to access the .git\r
+repository (from which you can still checkout a tree tying the blobs\r
+into a Maildir). The benefit would be compression, lower inode count\r
+(due to packs), and backups using clones/merges.\r
+\r
+You could either have the MDA write to a Git repo on the server side\r
+and use git packs to download mail to a local clone, or one could\r
+have e.g. offlineimap grow a Git storage backend. The interface to\r
+notmuch would be the same.\r
+\r
+If we used this, all the rename and delete code would be refactored\r
+into Git and could be removed from notmuch. In addition, notmuch\r
+could actually use Git tree objects to represent the results of\r
+searches, and you could checkout these trees. However, deleting\r
+messages from search results would not have any effect on the\r
+message or its existence in other search results, much like what\r
+happens with mairix nowadays.\r
+\r
+I think we all kinda agreed that the Maildir flags should not be\r
+used by notmuch and that things like Sebastian's notmuchsync should\r
+be used if people wanted flags represented in Maildir filenames.\r
+\r
+Instead of a Maildir checkout, notmuch could provide an interface to\r
+browse the store contents in a way that could make it accessible to\r
+mutt. The argument is that with 'notmuch {ls,cat,rm,=E2=80=A6}', a mutt\r
+backend could be trivially written. I am not sure about that, but\r
+it's worth a try.\r
+\r
+But there are still good reasons why you'd want to have IMAP\r
+capability too, e.g. Webmail. Given the atomicity problems that come\r
+=66rom Git, maybe an IMAP server reading from the Git store would make\r
+sense.\r
+\r
+However, this all sounds like a lot of NIH and reinvention. It's\r
+a bit like the marriage between the hypothetical Maildir2 and Git,\r
+which is definitely worth pursuing. Before we embark on any of this,\r
+however, we'd need to define the way in which Git stores mail.\r
+\r
+Stewart, you've worked most on this so far. Would you like to share\r
+your thoughts?\r
+\r
+--=20\r
+martin | http://madduck.net/ | http://two.sentenc.es/\r
+=20\r
+"reife des mannes, das ist es,\r
+ den ernst wiedergefunden zu haben, den\r
+ man hatte als kind beim spiel."\r
+                                                -- friedrich nietzsche\r
+=20\r
+spamtraps: madduck.bogus@madduck.net\r
+\r
+--fdj2RfSjLxBAspz7\r
+Content-Type: application/pgp-signature; name="digital_signature_gpg.asc"\r
+Content-Description: Digital signature (see http://martin-krafft.net/gpg/)\r
+Content-Disposition: inline\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.10 (GNU/Linux)\r
+\r
+iEYEAREDAAYFAktc6f0ACgkQIgvIgzMMSnXpgQCfSKQK+ZC5xqzoKKLq9ksNDln7\r
+58wAnittDhnDlFoZPLR8SwZDo5Q+ap2v\r
+=Vcry\r
+-----END PGP SIGNATURE-----\r
+\r
+--fdj2RfSjLxBAspz7--\r