From: Tomi Ollila <tomi.ollila@iki.fi>
To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>,
 David Bremner <david@tethera.net>, notmuch@notmuchmail.org
Subject: Re: [RFC2 Patch 5/5] lib: iterator API for message properties
In-Reply-To: <87pos1u14p.fsf@alice.fifthhorseman.net>
References: <1463927339-5441-1-git-send-email-david@tethera.net>
 <1464608999-14774-1-git-send-email-david@tethera.net>
 <1464608999-14774-6-git-send-email-david@tethera.net>
 <8760tthfuy.fsf@zancas.localnet> <87pos1u14p.fsf@alice.fifthhorseman.net>
User-Agent: Notmuch/0.22+32~gd4854c5 (http://notmuchmail.org) Emacs/24.5.1
 (x86_64-unknown-linux-gnu)
Date: Wed, 01 Jun 2016 08:04:17 +0300
Message-ID: <m2oa7lcxf2.fsf@guru.guru-group.fi>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list

On Wed, Jun 01 2016, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:

> On Tue 2016-05-31 21:12:21 -0400, David Bremner <david@tethera.net> wrote:
>> I was thinking a bit about how to dump/restore these.
>>
>> The most upwardly compatible way that i thought of is something like
>>
>> #=3D msg-id key=3Dval key=3Dval
>>
>> i.e. duplicate the msg-id for messages with properties
>>
>> This would be ignored by old notmuch-restore.
>>
>> Otherwise, maybe something like
>>
>> msg-id -- +tag +tag # key=3Dval key=3Dval
>>
>> I'm not sure. this might crash old notmuch-restore.
>>
>> How important is backward compatibility, and how important is minimizing
>> dump size? It's a bit hard to predict the things people might use
>> message properties for, but for thread surgery, I would expect a small
>> number of messages with properties.
>
> The other concern is our conception of how properties are unset/removed,
> right?
>
> With tags, it's possible to include -blah to remove the tag "blah".  how
> do we remove/clear/overwrite these tags?  what about using +key=3Dval or
> -key=3Dval to set/unset certain key/value combinations, and a value-less
> key=3D to remove all values matching a given key?
>
> alternately:
>
>  key=3Dval (clears all values for "key", and sets a new value "val")
>  key+=3Dval (appends a value "val" for "key")
>  key-=3Dval (removes any "key" set to "val")
>  key=3D (clears all values for "key"

We'd have to distinct between key being empty and unset,
comparable to how notmuch config behaves...


$ notmuch config get built_with.compact
true
$
$ notmuch config get search.exclude_tag
$
$ notmuch config get search.exclude_tagsz=20
Unknown configuration item: search.exclude_tagsz
zsh: exit 1     notmuch config get search.exclude_tagsz

>
> ---------
>
> However we resolve this particular decision, it'd be nice to have a
> stable, sane story about backward compatibility going forward, so that
> we don't have to worry about it in the future.
>
> For example, each dump file could start with a line like:
>
>   #version 1
>
> and notmuch restore would assume that without "#version n" as the first
> line, it's version 0.  then notmuch restore could decline to parse dump
> files of a version that it doesn't know about.
>
> Alternately, we could have the first line be something like:
>
>    #features config properties
>
> and if the first line is not #features, then we assume that no features
> are in place -- but if restore sees features it doesn't know about, it
> can offer to proceed while warning the user that we might miss something
> (or that something might break).

Currently dump output starts with (just run notmuch dump | less)

#notmuch-dump batch-tag:2 config,tags

perhaps this info could be put there -- is restore now (since a few notmuch
versions) already declining if this contains some strange data ?

of the 2 above suggestions I'd go w/ compatibilty version; it might be
challenging to get old notmuch parse relevant data from newer format...
... unless we also change the format to something more structured (jso^H^H^=
H^G
where only known data can be extracted (no, it is not SMOP, NO!)

> Thanks for working on this, David!  I think this is going to be really
> useful!

=C3=96h, what is this feature for... >;) maybe I have to look into the seri=
es
deeper...

>
>     --dkg

Tomi