Return-Path: <tils@tils.net>
X-Original-To: notmuch@notmuchmail.org
Delivered-To: notmuch@notmuchmail.org
Received: from localhost (localhost [127.0.0.1])
	by olra.theworths.org (Postfix) with ESMTP id 1B9BD431FBD
	for <notmuch@notmuchmail.org>; Sun, 13 Apr 2014 04:59:24 -0700 (PDT)
X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
X-Spam-Flag: NO
X-Spam-Score: 0
X-Spam-Level: 
X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]
	autolearn=disabled
Received: from olra.theworths.org ([127.0.0.1])
	by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id fQGnBjNSeDIT for <notmuch@notmuchmail.org>;
	Sun, 13 Apr 2014 04:59:20 -0700 (PDT)
X-Greylist: delayed 368 seconds by postgrey-1.32 at olra;
	Sun, 13 Apr 2014 04:59:20 PDT
Received: from goto.r-w-x.org (goto.r-w-x.org [176.9.70.178])
	by olra.theworths.org (Postfix) with ESMTP id B0A7F431FBC
	for <notmuch@notmuchmail.org>; Sun, 13 Apr 2014 04:59:20 -0700 (PDT)
Received: from localhost (55d456ec.access.ecotel.net [85.212.86.236])
	by goto.r-w-x.org (Postfix) with ESMTPSA id F0D64BF68;
	Sun, 13 Apr 2014 13:53:05 +0200 (CEST)
Received: by localhost (sSMTP sendmail emulation);
	Sun, 13 Apr 2014 13:53:05 +0200
From: Tilmann Singer <tils@tils.net>
To: Brian Sniffen <bsniffen@akamai.com>, notmuch@notmuchmail.org
Subject: Re: Synchronization success stories?
In-Reply-To: <m2r455f270.fsf@usma1mc-0csx92.kendall.corp.akamai.com>
References: <m2r455f270.fsf@usma1mc-0csx92.kendall.corp.akamai.com>
User-Agent: Notmuch/0.17 (http://notmuchmail.org) Emacs/24.3.1
	(x86_64-unknown-linux-gnu)
Date: Sun, 13 Apr 2014 13:53:01 +0200
Message-ID: <87ppklwin6.fsf@tils.net>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"
X-BeenThere: notmuch@notmuchmail.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: "Use and development of the notmuch mail system."
	<notmuch.notmuchmail.org>
List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
	<mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
List-Archive: <http://notmuchmail.org/pipermail/notmuch>
List-Post: <mailto:notmuch@notmuchmail.org>
List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
	<mailto:notmuch-request@notmuchmail.org?subject=subscribe>
X-List-Received-Date: Sun, 13 Apr 2014 11:59:24 -0000

--=-=-=
Content-Type: text/plain

I have experimented with offlineimap, syncmaildir and rsync.  The
append-only approach of notmuch makes synchronization of the mail corpus
simpler, so there are lots of options.  With ssh access to the server, I
found rsync to be conceptually the simplest, but it turned out to be too
slow for me (with ~110k mails) when frequently checking for new mails.

What I have settled with is a hacked together ruby script that uses the
notmuch command line both on the server and on the client to determine
unsynced mails, and then runs rsync explicitely for the necessary files.

The notmuch index on the server is only used to find new files for this
synchronization process, and is different from the notmuch indexes I
have on my client machines.

A prerequisite for this is of course ssh access and the ability to set
up notmuch on the server.

The steps performed on a sync run are roughly like this:

- local: notmuch new
- local: notmuch search --output=messages <some time ago>..<now>
- remote: notmuch new
- remote: notmuch search --output=messages <some time ago>..<now>
- compare search results
- run rsync for mails that only exist locally
  (using notmuch search --output=files to get the filenames)
- run rsync for mails that only exist remotely
  (using notmuch search --output=files to get the filenames)

With a reused ssh connection this is sufficiently fast for me (<2s).  If
there is interest I can clean up the script of hardcoded paths etc. and
put it on github.

Synchronization of the notmuch tags database is only necessary when I
switch between different client computers, which happens less
frequently. Like David I have a dump file committed to git for that. I
found it useful to sort the output before adding it to git, to avoid
huge unreadable diffs:

notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump


Til

--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJTSnqhAAoJEIPWI/RHW7lz6kkH/jSSYmzBRlV+jXPxfsBSSutA
w7aM0VX/BBgwz+OiiMKfJMHA8xS4Q7AnWojcimki14WlT6VuUsNxn0mqTo8/3GM3
ju27rHEvSH/sQUHnFgnoIAh1sGwP/IspKJc+225FVQPpZhe/kgv0RO/Bd2d9furi
fZCHsJy8mzVbLIeq3lCoy6H9jbJJrZTzl5UvRvYKuav78nucLAJMFnlpU57wLRuZ
BDY17rVOn6ZiToL8Mvizbrjm7EttXiezV8ghDllGtMfkc/Pwq0kYNYCxrN3/GYx6
3PJsuNqBxQJhb7VesbKRdRRO10b1tXh4bPb/PSGDadU8gZAPqUOIvoBlXxSwMH8=
=p3sn
-----END PGP SIGNATURE-----
--=-=-=--