Re: Synchronization success stories?

diff --git a/f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb b/f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb
new file mode 100644 (file)
index 0000000..5bace84
--- /dev/null
+++ b/f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb
@@ -0,0 +1,114 @@
+Return-Path: <tils@tils.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 1B9BD431FBD\r
+       for <notmuch@notmuchmail.org>; Sun, 13 Apr 2014 04:59:24 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id fQGnBjNSeDIT for <notmuch@notmuchmail.org>;\r
+       Sun, 13 Apr 2014 04:59:20 -0700 (PDT)\r
+X-Greylist: delayed 368 seconds by postgrey-1.32 at olra;\r
+       Sun, 13 Apr 2014 04:59:20 PDT\r
+Received: from goto.r-w-x.org (goto.r-w-x.org [176.9.70.178])\r
+       by olra.theworths.org (Postfix) with ESMTP id B0A7F431FBC\r
+       for <notmuch@notmuchmail.org>; Sun, 13 Apr 2014 04:59:20 -0700 (PDT)\r
+Received: from localhost (55d456ec.access.ecotel.net [85.212.86.236])\r
+       by goto.r-w-x.org (Postfix) with ESMTPSA id F0D64BF68;\r
+       Sun, 13 Apr 2014 13:53:05 +0200 (CEST)\r
+Received: by localhost (sSMTP sendmail emulation);\r
+       Sun, 13 Apr 2014 13:53:05 +0200\r
+From: Tilmann Singer <tils@tils.net>\r
+To: Brian Sniffen <bsniffen@akamai.com>, notmuch@notmuchmail.org\r
+Subject: Re: Synchronization success stories?\r
+In-Reply-To: <m2r455f270.fsf@usma1mc-0csx92.kendall.corp.akamai.com>\r
+References: <m2r455f270.fsf@usma1mc-0csx92.kendall.corp.akamai.com>\r
+User-Agent: Notmuch/0.17 (http://notmuchmail.org) Emacs/24.3.1\r
+       (x86_64-unknown-linux-gnu)\r
+Date: Sun, 13 Apr 2014 13:53:01 +0200\r
+Message-ID: <87ppklwin6.fsf@tils.net>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; boundary="=-=-=";\r
+       micalg=pgp-sha1; protocol="application/pgp-signature"\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 13 Apr 2014 11:59:24 -0000\r
+\r
+--=-=-=\r
+Content-Type: text/plain\r
+\r
+I have experimented with offlineimap, syncmaildir and rsync.  The\r
+append-only approach of notmuch makes synchronization of the mail corpus\r
+simpler, so there are lots of options.  With ssh access to the server, I\r
+found rsync to be conceptually the simplest, but it turned out to be too\r
+slow for me (with ~110k mails) when frequently checking for new mails.\r
+\r
+What I have settled with is a hacked together ruby script that uses the\r
+notmuch command line both on the server and on the client to determine\r
+unsynced mails, and then runs rsync explicitely for the necessary files.\r
+\r
+The notmuch index on the server is only used to find new files for this\r
+synchronization process, and is different from the notmuch indexes I\r
+have on my client machines.\r
+\r
+A prerequisite for this is of course ssh access and the ability to set\r
+up notmuch on the server.\r
+\r
+The steps performed on a sync run are roughly like this:\r
+\r
+- local: notmuch new\r
+- local: notmuch search --output=messages <some time ago>..<now>\r
+- remote: notmuch new\r
+- remote: notmuch search --output=messages <some time ago>..<now>\r
+- compare search results\r
+- run rsync for mails that only exist locally\r
+  (using notmuch search --output=files to get the filenames)\r
+- run rsync for mails that only exist remotely\r
+  (using notmuch search --output=files to get the filenames)\r
+\r
+With a reused ssh connection this is sufficiently fast for me (<2s).  If\r
+there is interest I can clean up the script of hardcoded paths etc. and\r
+put it on github.\r
+\r
+Synchronization of the notmuch tags database is only necessary when I\r
+switch between different client computers, which happens less\r
+frequently. Like David I have a dump file committed to git for that. I\r
+found it useful to sort the output before adding it to git, to avoid\r
+huge unreadable diffs:\r
+\r
+notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump\r
+\r
+\r
+Til\r
+\r
+--=-=-=\r
+Content-Type: application/pgp-signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v2.0.22 (GNU/Linux)\r
+\r
+iQEcBAEBAgAGBQJTSnqhAAoJEIPWI/RHW7lz6kkH/jSSYmzBRlV+jXPxfsBSSutA\r
+w7aM0VX/BBgwz+OiiMKfJMHA8xS4Q7AnWojcimki14WlT6VuUsNxn0mqTo8/3GM3\r
+ju27rHEvSH/sQUHnFgnoIAh1sGwP/IspKJc+225FVQPpZhe/kgv0RO/Bd2d9furi\r
+fZCHsJy8mzVbLIeq3lCoy6H9jbJJrZTzl5UvRvYKuav78nucLAJMFnlpU57wLRuZ\r
+BDY17rVOn6ZiToL8Mvizbrjm7EttXiezV8ghDllGtMfkc/Pwq0kYNYCxrN3/GYx6\r
+3PJsuNqBxQJhb7VesbKRdRRO10b1tXh4bPb/PSGDadU8gZAPqUOIvoBlXxSwMH8=\r
+=p3sn\r
+-----END PGP SIGNATURE-----\r
+--=-=-=--\r