From: David Mazieres <dm-list-email-notmuch@scs.stanford.edu>
To: Amadeusz =?utf-8?B?xbtvxYJub3dza2k=?= <aidecoe@aidecoe.name>,
 notmuch@notmuchmail.org
Subject: Re: muchsync files renames
In-Reply-To: <871teu8kdd.fsf@freja.aidecoe.name>
References: <878u93ujdo.fsf@freja.aidecoe.name>
 <876146o920.fsf@ta.scs.stanford.edu> <871teu8kdd.fsf@freja.aidecoe.name>
Reply-To: David Mazieres expires 2015-11-21 PST
 <mazieres-f72cdtyxmh2n26c2fbep99wgpw@temporary-address.scs.stanford.edu>
Date: Sun, 23 Aug 2015 13:06:20 -0700
Message-ID: <87oahxojlv.fsf@ta.scs.stanford.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list

Amadeusz =C5=BBo=C5=82nowski <aidecoe@aidecoe.name> writes:

> Hi David,
>
> Fist of all thank you for such elaborate answer.
>
> I have missed the paragraph about maildir.synchronize_flags somehow.  I
> have it enabled.  So this must be source of a problem (?).

I've only ever tested with mailder.synchronize_flags =3D true, because I'm
worried that if I disable it I will have a hard time re-enabling it.  I
do think it is a source of inefficiency, but muchsync should still be
usable.

> Here follows steps I followed:
>
> 1. I initialized server locally with muchsync -vv.  My mail is stored in
> ~/Mail on the server.
> 2. I run muchsync --init ~/mail SERVER. (Directory names do not need to
> be the same, do they?)

Confirmed that directory names do not need to be the same.

> 3. I run muchsync SERVER.
> 4. When it lasted much longer then initialization I canceled it by
> single SIGINT (^c).

Interesting.  I wish I knew why this was taking much longer than running
it on the server, and whether the delay was caused by client activity or
server activity.

I don't suppose you'd be willing to make a copy of your mail database to
repeat the experiment without any risk of messing up your real maildir?
Basically what would be interesting to see is (assuming .notmuch-copy on
server is configured for this disposable copy):

    # Log everything for later analysis
    script
    # Use new config file location for disposable copy
    export NOTMUCH_CONFIG=3D$HOME/.notmuch-copy
    # Set up a new initial database
    muchsync -vvvv --init ~/test-copy SERVER -vvvv --config=3D.notmuch-copy

    # Now initial copy is made, see if client is slow
    # Is notmuch new itself slow?
    notmuch new
    # Is resynchronizing muchsync with notmuch slow?
    muchsync -vvvv

    # Now see if something weird happened on server to make notmuch new slow
    ssh SERVER notmuch --config=3D.notmuch-copy new
    # Now see if something weird happened on server to make muchsync slow
    ssh SERVER muchsync -vvvv --config=3D.notmuch-copy

    # Now finally try resynchronizing to see if this is slow
    muchsync -vvvv SERVER -vvvv --config=3D.notmuch-copy
    # Save script file
    exit

Does something along these lines make sense?  If you can figure out
which of these is slow (other than --init, which always will be), and
what the verbose chatter is, that would help me narrow down and identify
any problem.

> 5. I rerun muchsync SERVER and then it notified me that notmuch
> identified files names changes - more than 1000.

Were the link changes on the client (sent) or the server (received)
side?

> 6. I waited a bit and then I canceled it by SIGINT.
> 7. I run muchsync --noup SERVER. This took only seconds to finish.

So this suggests the issue is on the client side.  It sounds like a bug.
I wonder if I we can just reproduce this using a public email corpus, so
we don't have to worry about people's private email.

> I suspected that muchsync at step 3 and 5 tried to push files renames
> back to server.  But now I am not sure what was going on.  Have I
> desynchronized file mail flags?  It's hard to say if anything has broken
> for me, but I am a bit worried anyway.
>
> If I just disable maildir.synchronize_flags and rerun muchsync, will
> everything get synchronized properly?

I don't think that will change things.  maildir.synchronized_flags will
make things slower, but it shouldn't affect the SUMMARY numbers you see
at the end of muchsync, other than maybe files moving from .../new to
.../cur.  But presumably most of your mail files were already in cur
directories.

David