From 5e01aa2afe4d4e1b68e76db4571c88e920ac9249 Mon Sep 17 00:00:00 2001 From: David Mazieres Date: Mon, 24 Aug 2015 13:06:20 +1700 Subject: [PATCH] Re: muchsync files renames --- 34/8939b308e764eda424b32f72aed575acfc69cc | 149 ++++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 34/8939b308e764eda424b32f72aed575acfc69cc diff --git a/34/8939b308e764eda424b32f72aed575acfc69cc b/34/8939b308e764eda424b32f72aed575acfc69cc new file mode 100644 index 000000000..8eef93094 --- /dev/null +++ b/34/8939b308e764eda424b32f72aed575acfc69cc @@ -0,0 +1,149 @@ +Return-Path: + +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by arlo.cworth.org (Postfix) with ESMTP id 4EB406DE13DB + for ; Sun, 23 Aug 2015 13:06:30 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at cworth.org +X-Spam-Flag: NO +X-Spam-Score: -2.261 +X-Spam-Level: +X-Spam-Status: No, score=-2.261 tagged_above=-999 required=5 tests=[AWL=0.590, + RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.55, SPF_PASS=-0.001] + autolearn=disabled +Received: from arlo.cworth.org ([127.0.0.1]) + by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id AVNoyFJljYQs for ; + Sun, 23 Aug 2015 13:06:28 -0700 (PDT) +Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10]) + by arlo.cworth.org (Postfix) with ESMTPS id 5C0D76DE13DA + for ; Sun, 23 Aug 2015 13:06:28 -0700 (PDT) +Received: from market.scs.stanford.edu (localhost.scs.stanford.edu + [127.0.0.1]) by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id + t7NK6LXZ024280; Sun, 23 Aug 2015 13:06:21 -0700 (PDT) +Received: (from dm@localhost) + by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id t7NK6KAv003545; + Sun, 23 Aug 2015 13:06:20 -0700 (PDT) +X-Authentication-Warning: market.scs.stanford.edu: dm set sender to + return-jmxwjhzeuidrb5in85caytaxb2@ta.scs.stanford.edu using -f +From: David Mazieres +To: Amadeusz =?utf-8?B?xbtvxYJub3dza2k=?= , + notmuch@notmuchmail.org +Subject: Re: muchsync files renames +In-Reply-To: <871teu8kdd.fsf@freja.aidecoe.name> +References: <878u93ujdo.fsf@freja.aidecoe.name> + <876146o920.fsf@ta.scs.stanford.edu> <871teu8kdd.fsf@freja.aidecoe.name> +Reply-To: David Mazieres expires 2015-11-21 PST + +Date: Sun, 23 Aug 2015 13:06:20 -0700 +Message-ID: <87oahxojlv.fsf@ta.scs.stanford.edu> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Transfer-Encoding: quoted-printable +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.18 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 23 Aug 2015 20:06:30 -0000 + +Amadeusz =C5=BBo=C5=82nowski writes: + +> Hi David, +> +> Fist of all thank you for such elaborate answer. +> +> I have missed the paragraph about maildir.synchronize_flags somehow. I +> have it enabled. So this must be source of a problem (?). + +I've only ever tested with mailder.synchronize_flags =3D true, because I'm +worried that if I disable it I will have a hard time re-enabling it. I +do think it is a source of inefficiency, but muchsync should still be +usable. + +> Here follows steps I followed: +> +> 1. I initialized server locally with muchsync -vv. My mail is stored in +> ~/Mail on the server. +> 2. I run muchsync --init ~/mail SERVER. (Directory names do not need to +> be the same, do they?) + +Confirmed that directory names do not need to be the same. + +> 3. I run muchsync SERVER. +> 4. When it lasted much longer then initialization I canceled it by +> single SIGINT (^c). + +Interesting. I wish I knew why this was taking much longer than running +it on the server, and whether the delay was caused by client activity or +server activity. + +I don't suppose you'd be willing to make a copy of your mail database to +repeat the experiment without any risk of messing up your real maildir? +Basically what would be interesting to see is (assuming .notmuch-copy on +server is configured for this disposable copy): + + # Log everything for later analysis + script + # Use new config file location for disposable copy + export NOTMUCH_CONFIG=3D$HOME/.notmuch-copy + # Set up a new initial database + muchsync -vvvv --init ~/test-copy SERVER -vvvv --config=3D.notmuch-copy + + # Now initial copy is made, see if client is slow + # Is notmuch new itself slow? + notmuch new + # Is resynchronizing muchsync with notmuch slow? + muchsync -vvvv + + # Now see if something weird happened on server to make notmuch new slow + ssh SERVER notmuch --config=3D.notmuch-copy new + # Now see if something weird happened on server to make muchsync slow + ssh SERVER muchsync -vvvv --config=3D.notmuch-copy + + # Now finally try resynchronizing to see if this is slow + muchsync -vvvv SERVER -vvvv --config=3D.notmuch-copy + # Save script file + exit + +Does something along these lines make sense? If you can figure out +which of these is slow (other than --init, which always will be), and +what the verbose chatter is, that would help me narrow down and identify +any problem. + +> 5. I rerun muchsync SERVER and then it notified me that notmuch +> identified files names changes - more than 1000. + +Were the link changes on the client (sent) or the server (received) +side? + +> 6. I waited a bit and then I canceled it by SIGINT. +> 7. I run muchsync --noup SERVER. This took only seconds to finish. + +So this suggests the issue is on the client side. It sounds like a bug. +I wonder if I we can just reproduce this using a public email corpus, so +we don't have to worry about people's private email. + +> I suspected that muchsync at step 3 and 5 tried to push files renames +> back to server. But now I am not sure what was going on. Have I +> desynchronized file mail flags? It's hard to say if anything has broken +> for me, but I am a bit worried anyway. +> +> If I just disable maildir.synchronize_flags and rerun muchsync, will +> everything get synchronized properly? + +I don't think that will change things. maildir.synchronized_flags will +make things slower, but it shouldn't affect the SUMMARY numbers you see +at the end of muchsync, other than maybe files moving from .../new to +.../cur. But presumably most of your mail files were already in cur +directories. + +David -- 2.26.2