--- /dev/null
+Return-Path: <wking@tremily.us>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 16C81431FBC\r
+ for <notmuch@notmuchmail.org>; Fri, 7 Nov 2014 11:05:32 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 3.181\r
+X-Spam-Level: ***\r
+X-Spam-Status: No, score=3.181 tagged_above=-999 required=5\r
+ tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,\r
+ FRT_SOMA=3.28, FRT_SOMA2=0.001, RCVD_IN_DNSWL_NONE=-0.0001]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id jK9x8K3STcqZ for <notmuch@notmuchmail.org>;\r
+ Fri, 7 Nov 2014 11:05:23 -0800 (PST)\r
+Received: from resqmta-po-03v.sys.comcast.net (resqmta-po-03v.sys.comcast.net\r
+ [96.114.154.162])\r
+ (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 85174431FB6\r
+ for <notmuch@notmuchmail.org>; Fri, 7 Nov 2014 11:05:23 -0800 (PST)\r
+Received: from resomta-po-02v.sys.comcast.net ([96.114.154.226])\r
+ by resqmta-po-03v.sys.comcast.net with comcast\r
+ id Cj4Y1p0084tLnxL01j5P4q; Fri, 07 Nov 2014 19:05:23 +0000\r
+Received: from odin.tremily.us ([24.18.63.50])\r
+ by resomta-po-02v.sys.comcast.net with comcast\r
+ id Cj3N1p005152l3L01j3NKA; Fri, 07 Nov 2014 19:03:23 +0000\r
+Received: by odin.tremily.us (Postfix, from userid 1000)\r
+ id AD3631476F60; Fri, 7 Nov 2014 11:03:21 -0800 (PST)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tremily.us; s=odin;\r
+ t=1415387001; bh=Kh2QLEy4SAP5cx4l1iaezk+MrusS9fnXrUxa4FYdJUw=;\r
+ h=Date:From:To:Cc:Subject;\r
+ b=QlDq+Pvb+F9u6VN6FLPdEWkSgOglKj3QK0fJsxfGKjZVN8QQnpuO4JiXpB9qbIZIF\r
+ PnE0EJxcMQdel+3d6QF7WQvKInR/bIK/juQ87buJPerXtam+lZ8GEXNeeGqiuLqWvI\r
+ xQ6roGlXAr7i2KUzn+1++O9nU8pock1SIv4H54MY=\r
+Date: Fri, 7 Nov 2014 11:03:21 -0800\r
+From: "W. Trevor King" <wking@tremily.us>\r
+To: notmuch@notmuchmail.org\r
+Subject: Mail archives in Git using ssoma\r
+Message-ID: <20141107190321.GL23609@odin.tremily.us>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; micalg=pgp-sha1;\r
+ protocol="application/pgp-signature"; boundary="9JSHP372f+2dzJ8X"\r
+Content-Disposition: inline\r
+OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8;\r
+ url=http://tremily.us/pubkey.txt\r
+User-Agent: Mutt/1.5.23 (2014-03-12)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net;\r
+ s=q20140121; t=1415387123;\r
+ bh=EMVFRZ821lxzwm/G4cwidMqYECm8nLl+PTjsH7YQgfc=;\r
+ h=Received:Received:Received:Date:From:To:Subject:Message-ID:\r
+ MIME-Version:Content-Type;\r
+ b=maAC+hXiJiVeTj+ewX3r21eF6EyrELH1QknYcT5nmUHSuhMpMMdA0xOnjxkgybAmt\r
+ ZoS1qi6ELkZTJ820eHxeIQ46z4pGBnrUyGlnDbD5Jgz3QzZC+90ynnk1kO5AcvOXcF\r
+ zEqeO6VkgseP4KZVAn3aQyF8uZVI6EioFxohztvVXcqk1jKlguia3S1+XmmV5z5sh3\r
+ UYwEaEj5QSHfMnBnCKu5NkJtlb1guqsidWYJNFcY6x8lipa+Gqi9uS26uqhEcU2zzA\r
+ cQeBzdr1WztR1Ra63FOaRAG2Vx+YQLMdDEu20L2HFhm6wbt9S0PySCIuSc+6toMebB\r
+ /+n/87/eNSakg==\r
+Cc: Eric Wong <e@80x24.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 07 Nov 2014 19:05:32 -0000\r
+\r
+\r
+--9JSHP372f+2dzJ8X\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: quoted-printable\r
+\r
+Hello everyone :),\r
+\r
+I like Git, so when folks suggest storing things in Git, I'm usually\r
+excited ;). Eric Wong has been working on some tools to store email\r
+in a Git repository, and his client-side code is ssoma [1]. I wanted\r
+a bit more metadata than the stock ssoma-mda [2], and ended up just\r
+writing a ssoma-mda in Python [3]. It needs Python =E2=89=A53.4 and pygit2.\r
+I had pygit2 already installed for Python 3.3 (which gave me a local\r
+libgit2), so I used pip to install it for 3.4:\r
+\r
+ $ python3.4 -m ensurepip --user\r
+ $ pip3.4 install --user pygit2\r
+\r
+Then I grabbed the archives, and pulled them into Git:\r
+\r
+ $ wget http://notmuchmail.org/archives/notmuch.mbox\r
+ $ git init --bare notmuch-archives.git\r
+ $ cd notmuch-archives.git\r
+ $ python3.4\r
+ >>> import email.utils\r
+ >>> import mailbox\r
+ >>> import ssoma_mda\r
+ >>> mbox =3D mailbox.mbox('../notmuch.mbox', factory=3DNone, create=3DFal=\r
+se)\r
+ >>> messages =3D sorted(mbox, key=3Dlambda m: email.utils.mktime_tz(email=\r
+=2Eutils.parsedate_tz(m['date'])))\r
+ >>> for message in messages:\r
+ ... if ((message['message-id'] =3D=3D '<m2k4gmyjer.fsf@ecocode.net>' =\r
+and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Sat, 26 Feb 2011 =\r
+14:23:34 -0000') or\r
+ ... (message['message-id'] =3D=3D '<4EDF728E.3050204@gmail.com>=\r
+' and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Wed, 07 Dec 2011 =\r
+14:05:16 -0000') or\r
+ ... (message['message-id'] =3D=3D <4FE369F2.5080804@gmail.com>'=\r
+ and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Thu, 21 Jun 2012 =\r
+18:38:07 -0000') or\r
+ ... (message['message-id'] =3D=3D '<5122353D.4060601@gmail.com>=\r
+' and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Mon, 18 Feb 2013 =\r
+14:06:12 -0000') or\r
+ ... (message['message-id'] =3D=3D '<CA+eQo_1hMsTD4+6ifqgEQXW0_q=\r
+YXGOdfkO6tBuGQKV+W7OSaKA@mail.gmail.com>' and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Wed, 24 Apr 2013 =\r
+18:09:55 -0000') or\r
+ ... (message['message-id'] =3D=3D '<527B9E8C.5000001@krugs.de>'=\r
+ and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Thu, 07 Nov 2013 =\r
+14:07:32 -0000') or\r
+ ... (message['message-id'] =3D=3D '<1399645162-8653-1-git-send-=\r
+email-wael.nasreddine@gmail.com>' and\r
+ ... message['X-List-Received-Date'] =3D=3D 'Fri, 09 May 2014 =\r
+14:19:36 -0000') or\r
+ ... (message['message-id'] =3D=3D '<m2mw9xkyvg.fsf@krugs.de>' a=\r
+nd\r
+ ... message['X-List-Received-Date'] =3D=3D 'Thu, 18 Sep 2014 =\r
+10:27:35 -0000') or\r
+ ... (message['message-id'] =3D=3D '<cover.1411379395.git.jani@n=\r
+ikula.org>' and\r
+ ... message['X-List-Received-Date'] !=3D 'Mon, 22 Sep 2014 09=\r
+:54:16 -0000')):\r
+ ... continue\r
+ ... ssoma_mda.deliver(message=3Dmessage, once=3DTrue)\r
+ >>> ^D\r
+\r
+On my 1.1GHz Intel Celeron 847 Sandy Bridge netbook, that took about\r
+half an hour. The initial repository was large:\r
+\r
+ $ du -hs .\r
+ 394M .\r
+\r
+But packing it up made it small:\r
+\r
+ $ git gc --aggressive\r
+ du -hs .\r
+ 51M .\r
+\r
+With a few less images than the mbox:\r
+\r
+ $ git log --oneline | wc -l\r
+ 19650\r
+\r
+Compared with 19660 messages in the mbox at 107 MB (160 MB for the\r
+associated Maildir).\r
+\r
+The messages I dropped removed duplicate Message-IDs:\r
+\r
+* id:m2k4gmyjer.fsf@ecocode.net had different received dates:\r
+\r
+ -X-List-Received-Date: Sat, 26 Feb 2011 14:12:20 -0000\r
+ +X-List-Received-Date: Sat, 26 Feb 2011 14:23:34 -0000\r
+\r
+ but no significant differences.\r
+\r
+* id:4EDF728E.3050204@gmail.com had a real address in the\r
+ first-to-arrive version:\r
+\r
+ -X-List-Received-Date: Wed, 07 Dec 2011 14:10:13 -0000\r
+ -> <4winter@informatik.uni-hamburg.de>\r
+\r
+ an an obfuscated one in the second-to-arrive version:\r
+\r
+ +X-List-Received-Date: Wed, 07 Dec 2011 14:05:16 -0000\r
+ +> <4winter-jNDFPZUTrfQBEfOqpokbeYV0Y/DQsy6Ps0AfqQuZ5sE@public.gmane.or=\r
+g>\r
+\r
+* id:4FE369F2.5080804@gmail.com had the same:\r
+\r
+ -X-List-Received-Date: Thu, 21 Jun 2012 18:37:54 -0000\r
+ -> <R.M.Krug@gmail.com\r
+\r
+ +X-List-Received-Date: Thu, 21 Jun 2012 18:38:07 -0000\r
+ -> <mailto:R.M.Krug@gmail.com>> wrote:\r
+\r
+* id:5122353D.4060601@gmail.com had different received dates:\r
+\r
+ -X-List-Received-Date: Mon, 18 Feb 2013 14:06:05 -0000\r
+ +X-List-Received-Date: Mon, 18 Feb 2013 14:06:12 -0000\r
+\r
+ but no significant differences.\r
+\r
+* id:CA+eQo_1hMsTD4+6ifqgEQXW0_qYXGOdfkO6tBuGQKV+W7OSaKA@mail.gmail.com\r
+ had different MIME boundaries:\r
+\r
+ -Content-Type: multipart/alternative; boundary=3Df46d043be11ac45a0904db=\r
+1f3428\r
+ -X-List-Received-Date: Wed, 24 Apr 2013 18:09:46 -0000\r
+\r
+ +Content-Type: multipart/alternative; boundary=3De89a8f646ff3faa11d04db=\r
+1f3294\r
+ +X-List-Received-Date: Wed, 24 Apr 2013 18:09:55 -0000\r
+\r
+ but no significant differences.\r
+\r
+* id:527B9E8C.5000001@krugs.de had obfuscated addresses:\r
+\r
+ -X-List-Received-Date: Thu, 07 Nov 2013 14:07:33 -0000\r
+ -> Rainer M Krug <Rainer@krugs.de> writes:\r
+\r
+ +X-List-Received-Date: Thu, 07 Nov 2013 14:07:32 -0000\r
+ +> Rainer M Krug <Rainer-vfylz/Ys1k4@public.gmane.org> writes:\r
+\r
+* id:1399645162-8653-1-git-send-email-wael.nasreddine@gmail.com had\r
+ additional content in the later submission:\r
+\r
+ -Subject: [PATCH] Add Travis-CI config file.\r
+ -Date: Fri, 9 May 2014 07:19:22 -0700\r
+ -X-List-Received-Date: Fri, 09 May 2014 14:19:36 -0000\r
+ - .travis.yml | 10 ++++++++++\r
+ - 1 file changed, 10 insertions(+)\r
+\r
+ +Subject: [PATCH v2] Enable Travis-CI as a backup continuous integration\r
+ + service.\r
+ +Date: Fri, 9 May 2014 14:44:50 -0700\r
+ +X-List-Received-Date: Fri, 09 May 2014 21:45:16 -0000\r
+ +\r
+ +The v2 adds a notification section to send failure (or back to passing=\r
+) notifications\r
+ +to the mailing list and to the IRC channel\r
+ +\r
+ + .travis.yml | 13 +++++++++++++\r
+ + 1 file changed, 13 insertions(+)\r
+\r
+* id:m2mw9xkyvg.fsf@krugs.de had an obfuscated adderss and different signat=\r
+ure:\r
+\r
+ -X-List-Received-Date: Thu, 18 Sep 2014 10:27:31 -0000\r
+ ->> guyzmo <guyzmo@m0g.net> writes:\r
+ -----BEGIN PGP SIGNATURE-----\r
+ Version: GnuPG/MacGPG2 v2.0.22 (Darwin)\r
+ -iQEcBAEBAgAGBQJUGrN3AAoJENvXNx4PUvmC4J0IAN9Wf+0ArvirJCoewItnEZoo\r
+ -ySg4VRP7uWVqDxHVl5N9XFv4YE2bZ2E2eMGvbo6v7I82lhqeR5dauZhlgCMki+ZI\r
+\r
+ +X-List-Received-Date: Thu, 18 Sep 2014 10:27:35 -0000\r
+ +>> guyzmo <guyzmo-kMjww5mZloE@public.gmane.org> writes:\r
+ -----BEGIN PGP SIGNATURE-----\r
+ Version: GnuPG/MacGPG2 v2.0.22 (Darwin)\r
+ +iQEcBAEBAgAGBQJUGrN4AAoJENvXNx4PUvmC6LsIAIaFrd4MFnm8EixrAHPGfW6j\r
+ +L3KNG7Dv+hQuNRUN6qn+emZHI8wX4O74HOZOpZWkE09CmjkPJBmf7IuJwtz2ONbM\r
+\r
+* id:cover.1411379395.git.jani@nikula.org came in three times, with\r
+ three dates, but no significant differences:\r
+\r
+ Date: Mon, 22 Sep 2014 11:54:20 +0200\r
+ X-List-Received-Date: Mon, 22 Sep 2014 09:54:16 -0000\r
+\r
+ Date: Mon, 22 Sep 2014 11:54:42 +0200\r
+ X-List-Received-Date: Mon, 22 Sep 2014 09:54:37 -0000\r
+\r
+ Date: Mon, 22 Sep 2014 11:54:51 +0200\r
+ X-List-Received-Date: Mon, 22 Sep 2014 09:54:49 -0000\r
+\r
+Anyhow, I've pushed the Git archive [4,5] if anyone wants to play\r
+around with ssoma. I think this would be a nice backend for folks\r
+building notmuch-based web archives, and pulling from Git is easier\r
+than downloading a new mbox ;).\r
+\r
+Cheers,\r
+Trevor\r
+\r
+[1]: http://ssoma.public-inbox.org/README\r
+[2]: http://public-inbox.org/meta/m/ec8f54cf6451eef6e9f59eff691cd9002f4fdf6=\r
+5.html\r
+[3]: http://git.tremily.us/?p=3Dssoma-mda.git;a=3Dshortlog;h=3Drefs/heads/p=\r
+ython\r
+ I have an uncommitted patch to work around http://bugs.python.org/issu=\r
+e22684\r
+[4]: http://git.tremily.us/?p=3Dnotmuch-archives.git\r
+[5]: git://tremily.us/notmuch-archives.git\r
+\r
+--=20\r
+This email may be signed or encrypted with GnuPG (http://www.gnupg.org).\r
+For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy\r
+\r
+--9JSHP372f+2dzJ8X\r
+Content-Type: application/pgp-signature; name="signature.asc"\r
+Content-Description: OpenPGP digital signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v2\r
+\r
+iQIcBAEBAgAGBQJUXRd3AAoJEG8/JgBt8ol86+IQAIJCbVS05SYDLgtTfo7+petJ\r
+hptvLRSWq0jNefa5r1ipOpCrzpWDSe/Rjcdb09jodvZzsNoHrS3rNQJPKmEvcE6b\r
+xP6YZ9wukBtMbSHx0XoCUpRri5LlQ2+774vMq2riT1X0qZov/uNG19XUorAoGj5U\r
+f64pYH7Q8rVk7NwfszNgmgbrujXoBMRIJV5CVkdiCOnTPxr9zmZc7wXuQheIueO6\r
+Ow0aR9A++Wo8lUwCpQPRqTr2Fl4xxBwtLhigJOezOh3gbqGavaua6j0K+B1oQ1nL\r
+W0iyE+GE4HVzx3npYWEqROMPnZ7Dsoiz2oQrbAZ+Xnkjw2SZyaFoI7KfpDa6WgD0\r
+hmVEdUBYD5uvrqmqKA12R6P70skiuujgKiW8npVcU2Xggoe0sS/gR6adkV2joF7F\r
+qeTNJ+AqzL2S7WQ7Kja43Y+a2Nrsk3nbDMDRmgUK+DL2JzXKcx9HtZO/9JeKMwh5\r
+xtsZJ08D2rgOMgM4pW6ZxZGcDLVeKDqvDF+dZA6v/ruaIJmbyen6RBGc6J63cSGI\r
+wfn1xFUbG0ZxhnV896UTuEMH5861pzenpXM2IZsT7T0XPCO/bTNdaBylnahQvBP4\r
+tIFD2smexq6CGAyw1SEy3CcJrFFyozAJ48gGaBmOdLt+SfoKrF9j/XX1bb4YqcWb\r
+q9xzd66reO3ffkkXPPuV\r
+=aswS\r
+-----END PGP SIGNATURE-----\r
+\r
+--9JSHP372f+2dzJ8X--\r
projects / notmuch-archives.git / commitdiff