From def7419bfcceb204c2c06c335a59ff5b09090a4f Mon Sep 17 00:00:00 2001 From: Tilmann Singer Date: Sun, 13 Apr 2014 13:53:01 +0200 Subject: [PATCH] Re: Synchronization success stories? --- f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb | 114 ++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb diff --git a/f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb b/f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb new file mode 100644 index 000000000..5bace8487 --- /dev/null +++ b/f7/2b15287c6e8c0cd9d8c44eae04f46cc3f9adbb @@ -0,0 +1,114 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 1B9BD431FBD + for ; Sun, 13 Apr 2014 04:59:24 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id fQGnBjNSeDIT for ; + Sun, 13 Apr 2014 04:59:20 -0700 (PDT) +X-Greylist: delayed 368 seconds by postgrey-1.32 at olra; + Sun, 13 Apr 2014 04:59:20 PDT +Received: from goto.r-w-x.org (goto.r-w-x.org [176.9.70.178]) + by olra.theworths.org (Postfix) with ESMTP id B0A7F431FBC + for ; Sun, 13 Apr 2014 04:59:20 -0700 (PDT) +Received: from localhost (55d456ec.access.ecotel.net [85.212.86.236]) + by goto.r-w-x.org (Postfix) with ESMTPSA id F0D64BF68; + Sun, 13 Apr 2014 13:53:05 +0200 (CEST) +Received: by localhost (sSMTP sendmail emulation); + Sun, 13 Apr 2014 13:53:05 +0200 +From: Tilmann Singer +To: Brian Sniffen , notmuch@notmuchmail.org +Subject: Re: Synchronization success stories? +In-Reply-To: +References: +User-Agent: Notmuch/0.17 (http://notmuchmail.org) Emacs/24.3.1 + (x86_64-unknown-linux-gnu) +Date: Sun, 13 Apr 2014 13:53:01 +0200 +Message-ID: <87ppklwin6.fsf@tils.net> +MIME-Version: 1.0 +Content-Type: multipart/signed; boundary="=-=-="; + micalg=pgp-sha1; protocol="application/pgp-signature" +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 13 Apr 2014 11:59:24 -0000 + +--=-=-= +Content-Type: text/plain + +I have experimented with offlineimap, syncmaildir and rsync. The +append-only approach of notmuch makes synchronization of the mail corpus +simpler, so there are lots of options. With ssh access to the server, I +found rsync to be conceptually the simplest, but it turned out to be too +slow for me (with ~110k mails) when frequently checking for new mails. + +What I have settled with is a hacked together ruby script that uses the +notmuch command line both on the server and on the client to determine +unsynced mails, and then runs rsync explicitely for the necessary files. + +The notmuch index on the server is only used to find new files for this +synchronization process, and is different from the notmuch indexes I +have on my client machines. + +A prerequisite for this is of course ssh access and the ability to set +up notmuch on the server. + +The steps performed on a sync run are roughly like this: + +- local: notmuch new +- local: notmuch search --output=messages .. +- remote: notmuch new +- remote: notmuch search --output=messages .. +- compare search results +- run rsync for mails that only exist locally + (using notmuch search --output=files to get the filenames) +- run rsync for mails that only exist remotely + (using notmuch search --output=files to get the filenames) + +With a reused ssh connection this is sufficiently fast for me (<2s). If +there is interest I can clean up the script of hardcoded paths etc. and +put it on github. + +Synchronization of the notmuch tags database is only necessary when I +switch between different client computers, which happens less +frequently. Like David I have a dump file committed to git for that. I +found it useful to sort the output before adding it to git, to avoid +huge unreadable diffs: + +notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump + + +Til + +--=-=-= +Content-Type: application/pgp-signature + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v2.0.22 (GNU/Linux) + +iQEcBAEBAgAGBQJTSnqhAAoJEIPWI/RHW7lz6kkH/jSSYmzBRlV+jXPxfsBSSutA +w7aM0VX/BBgwz+OiiMKfJMHA8xS4Q7AnWojcimki14WlT6VuUsNxn0mqTo8/3GM3 +ju27rHEvSH/sQUHnFgnoIAh1sGwP/IspKJc+225FVQPpZhe/kgv0RO/Bd2d9furi +fZCHsJy8mzVbLIeq3lCoy6H9jbJJrZTzl5UvRvYKuav78nucLAJMFnlpU57wLRuZ +BDY17rVOn6ZiToL8Mvizbrjm7EttXiezV8ghDllGtMfkc/Pwq0kYNYCxrN3/GYx6 +3PJsuNqBxQJhb7VesbKRdRRO10b1tXh4bPb/PSGDadU8gZAPqUOIvoBlXxSwMH8= +=p3sn +-----END PGP SIGNATURE----- +--=-=-=-- -- 2.26.2