From 72915077734de444df1b11e2923853d464115218 Mon Sep 17 00:00:00 2001 From: Gaute Hope Date: Tue, 23 Sep 2014 08:57:08 +0200 Subject: [PATCH] Re: [PATCH] Add configurable changed tag to messages that have been changed on disk --- 0b/699ccee986161690564cc98df0ef1343523060 | 308 ++++++++++++++++++++++ 1 file changed, 308 insertions(+) create mode 100644 0b/699ccee986161690564cc98df0ef1343523060 diff --git a/0b/699ccee986161690564cc98df0ef1343523060 b/0b/699ccee986161690564cc98df0ef1343523060 new file mode 100644 index 000000000..5db7a3430 --- /dev/null +++ b/0b/699ccee986161690564cc98df0ef1343523060 @@ -0,0 +1,308 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 21210431FAF + for ; Mon, 22 Sep 2014 23:57:26 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.699 +X-Spam-Level: +X-Spam-Status: No, score=-0.699 tagged_above=-999 required=5 + tests=[HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id cfyHJiruYHlt for ; + Mon, 22 Sep 2014 23:57:09 -0700 (PDT) +Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com + [209.85.192.44]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 89EF1431FAE + for ; Mon, 22 Sep 2014 23:57:09 -0700 (PDT) +Received: by mail-qg0-f44.google.com with SMTP id f51so4177176qge.3 + for ; Mon, 22 Sep 2014 23:57:08 -0700 (PDT) +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=1e100.net; s=20130820; + h=x-gm-message-state:mime-version:in-reply-to:references:date + :message-id:subject:from:to:cc:content-type; + bh=bqs7RpSw4T/jZaRqde4Q+ZnDMqzMYWouvanyODghIHQ=; + b=SH3XN7UZtLXTHZGzPyrja8W79mdxkdQxUk1GdKEPkMS5BzJxNpfr8jqw+QDhm65L3j + LdesHP4GnJZ5u0IH2y+0QV6Nq0E9jl4btob55FNuZ7WrN7m2/16WGYcNAFe0DgHmUxo4 + gQyceNWbB7VL5+JwAYjK+uQfRC2pcrA405wdcFBuD6Dz3tirDDnNt+YkKJ6yoAzBJL9t + wa5v+NF0tZOCfXiEJVSQeI7xXQJWMydXRx+1PJp6PioCVb/73ZSZ/1jj+Y06lX5d9wDO + rR1/AgsHhapCun/IqB9epszmzgrkDFvPmLvq6DWhzP6AC16+oPjtclKSVVupMfl4MEPP + UDlA== +X-Gm-Message-State: + ALoCoQmbhPmdvmHEI1JoECLSQ1OqSgnlGyjHpraZdrZV5u3ipXaKdTmkXdQxilJlP2MBE7Mtu0Js +MIME-Version: 1.0 +X-Received: by 10.224.46.6 with SMTP id h6mr29733002qaf.45.1411455428701; Mon, + 22 Sep 2014 23:57:08 -0700 (PDT) +Received: by 10.96.174.104 with HTTP; Mon, 22 Sep 2014 23:57:08 -0700 (PDT) +Received: by 10.96.174.104 with HTTP; Mon, 22 Sep 2014 23:57:08 -0700 (PDT) +In-Reply-To: <87k34vackm.fsf@drake.dyndns.org> +References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com> + <87fviiiuzn.fsf@maritornes.cs.unb.ca> + + <20140801185505.GS13893@mit.edu> + <1407313144-astroid-0-vyhth1tcrd-3835@strange> + <1411386960-astroid-2-k1e726ut3f-2518@strange> + <87k34vackm.fsf@drake.dyndns.org> +Date: Tue, 23 Sep 2014 08:57:08 +0200 +Message-ID: + +Subject: Re: [PATCH] Add configurable changed tag to messages that have been + changed on disk +From: Gaute Hope +To: Austin Clements +Content-Type: multipart/alternative; boundary=001a11c35a9c71cb080503b613b8 +Cc: notmuch +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 23 Sep 2014 06:57:26 -0000 + +--001a11c35a9c71cb080503b613b8 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: quoted-printable + +22. sep. 2014 17:40 skrev "Austin Clements" +f=C3=B8lgende: +> +> On Mon, 22 Sep 2014, Gaute Hope wrote: +> > Excerpts from Gaute Hope's message of August 6, 2014 10:29: +> >> Austin Clements wrote on Fri, 01 Aug 2014 14:55:05 +-0400: +> >>> I have a prototype implementation of message modification times on my +> >>> lastmod-v1 branch at +> >>> +> >>> https://github.com/aclements/notmuch/tree/lastmod-v1 +> >>> +> >>> It builds on my database features series that's currently awaiting +> >>> review [1]. +> >>> +> >>> The series uses a monotonic revision number, rather than wall-clock +> >>> time, for reasons related to Xapian's concurrent control and detailed +> >>> in the main commit's commit message. The implementation isn't quite +> >>> useful from the CLI yet because I haven't added any way to query the +> >>> database's current revision number. (I'm still thinking about how I +> >>> want to do this, since search/show don't have a good way to deliver +> >>> "additional" information right now. I might just add the last +> >>> modification for each individual message/max of all messages in a +> >>> thread, similar to what Thomas Jost's patch did long ago.) +> >>> +> >>> [1] id:1406859003-11561-1-git-send-email-amdragon@mit.edu +> > +> >> this should allow me to do what I wish to accomplish. The message +> >> deletion is still a problem though, I can see two options at the +moment: +> > +> > Hi list, +> > +> > While exploring the possibility of syncing maildir/X-keywords with tags +> > I had some thoughts about lastmod and message modification: +> > +> > As briefly discussed on #notmuch, I noticed that it seems that 'notmuch +> > new' does not detect that a message source has been changed, unless the +> > file is also re-named. +> > +> > This means that for instance if the X-Keywords fields have been updated +> > in a message (from GMail with offlineimap, synclabels =3D yes) the last= +mod +> > field will remain unchanged, and a source modification will be +> > undetectable to a client program using this value. +> > +> > Would it not make sense that if a message has a more recent mtime than +> > at index time it is re-indexed? +> +> This has the potential to make notmuch new substantially more expensive. +> Currently, if there are no changes, it only has to stat each directory +> in your maildir (in fact, some restructuring of new would let us +> eliminate almost all database access during a no-op notmuch new as +> well). Checking for changes to individual messages would require +> stat'ing every single message file as well as accessing the database to +> check the paths and mtimes of every message, increasing the number of +> stat calls and disk accesses by several orders of magnitude. +> +> It may be that this is fast enough that it's okay, but it would be good +> to gather some evidence first. That includes hot and cold caches, and +> maildir over NFS. +> +> With respect to X-Keywords specifically, note that it's a fairly basic +> design decision that notmuch never modifies message files. This gives +> us strong robustness guarantees we would be loathe to part with. +> +> It has puzzled me ever since offlineimap added X-Keywords why they +> didn't just translate these keywords into folders and create hard links +> of message files. Anything could interact smoothly with that. + +The information follows the message file. But, yeah, working directly on +the message source is hairy. Anyway, email is as mess in general anyway. I +consider it user-input. + +> +> > Also, for the lastmod branch I would wish for a notmuch_message_touch() +> > method where the lastmod time is updated to the last. As well as a +> > notmuch_database_reindex_message () - possibly defined/documented +> > behaviour for notmuch_database_add_message () when the filename is +> > already added (in which case I would expect notmuch to re-index the +> > message). +> +> What's the use case for these? + +If you make a change to the message source and want it to be reindexed. +Say, edited a draft or changed a header field. I am not asking that notmuch +modifies the message source. + +For _touch, if without making an actual change to the message you wish to +indicate that it has been updated or synced at the current time. For +instance after an reindex that did not make any actual change. Perhaps not +strictly necessary. + +Cheers, Gaute + +--001a11c35a9c71cb080503b613b8 +Content-Type: text/html; charset=UTF-8 +Content-Transfer-Encoding: quoted-printable + +


+22. sep. 2014 17:40 skrev "Austin Clements" <aclements@csail.mit.edu> f=C3=B8lgende: +>
+> On Mon, 22 Sep 2014, Gaute Hope <eg@gaute.vetsj.com> wrote:
+> > Excerpts from Gaute Hope's message of August 6, 2014 10:29: +> >> Austin Clements <amdra= +gon@MIT.EDU> wrote on Fri, 01 Aug 2014 14:55:05 -0400:
+> >>> I have a prototype implementation of message modification= + times on my
+> >>> lastmod-v1 branch at
+> >>>
+> >>>=C2=A0 =C2=A0https://github.com/aclements/notmuch/tree/lastmod-v1
+> >>>
+> >>> It builds on my database features series that's curre= +ntly awaiting
+> >>> review [1].
+> >>>
+> >>> The series uses a monotonic revision number, rather than = +wall-clock
+> >>> time, for reasons related to Xapian's concurrent cont= +rol and detailed
+> >>> in the main commit's commit message.=C2=A0 The implem= +entation isn't quite
+> >>> useful from the CLI yet because I haven't added any w= +ay to query the
+> >>> database's current revision number.=C2=A0 (I'm st= +ill thinking about how I
+> >>> want to do this, since search/show don't have a good = +way to deliver
+> >>> "additional" information right now.=C2=A0 I mig= +ht just add the last
+> >>> modification for each individual message/max of all messa= +ges in a
+> >>> thread, similar to what Thomas Jost's patch did long = +ago.)
+> >>>
+> >>> [1] id:1406859003-11561-1-git-send-email-amdragon@mit.edu= +
+> >
+> >> this should allow me to do what I wish to accomplish. The mes= +sage
+> >> deletion is still a problem though, I can see two options at = +the moment:
+> >
+> > Hi list,
+> >
+> > While exploring the possibility of syncing maildir/X-keywords wit= +h tags
+> > I had some thoughts about lastmod and message modification:
+> >
+> > As briefly discussed on #notmuch, I noticed that it seems that &#= +39;notmuch
+> > new' does not detect that a message source has been changed, = +unless the
+> > file is also re-named.
+> >
+> > This means that for instance if the X-Keywords fields have been u= +pdated
+> > in a message (from GMail with offlineimap, synclabels =3D yes) th= +e lastmod
+> > field will remain unchanged, and a source modification will be +> > undetectable to a client program using this value.
+> >
+> > Would it not make sense that if a message has a more recent mtime= + than
+> > at index time it is re-indexed?
+>
+> This has the potential to make notmuch new substantially more expensiv= +e.
+> Currently, if there are no changes, it only has to stat each directory= +
+> in your maildir (in fact, some restructuring of new would let us
+> eliminate almost all database access during a no-op notmuch new as
+> well).=C2=A0 Checking for changes to individual messages would require= +
+> stat'ing every single message file as well as accessing the databa= +se to
+> check the paths and mtimes of every message, increasing the number of<= +br> +> stat calls and disk accesses by several orders of magnitude.
+>
+> It may be that this is fast enough that it's okay, but it would be= + good
+> to gather some evidence first.=C2=A0 That includes hot and cold caches= +, and
+> maildir over NFS.
+>
+> With respect to X-Keywords specifically, note that it's a fairly b= +asic
+> design decision that notmuch never modifies message files.=C2=A0 This = +gives
+> us strong robustness guarantees we would be loathe to part with.
+>
+> It has puzzled me ever since offlineimap added X-Keywords why they
+> didn't just translate these keywords into folders and create hard = +links
+> of message files.=C2=A0 Anything could interact smoothly with that. +

The information follows the message file. But, yeah, working= + directly on the message source is hairy. Anyway, email is as mess in gener= +al anyway. I consider it user-input.

+

>
+> > Also, for the lastmod branch I would wish for a notmuch_message_t= +ouch()
+> > method where the lastmod time is updated to the last. As well as = +a
+> > notmuch_database_reindex_message () - possibly defined/documented= +
+> > behaviour for notmuch_database_add_message () when the filename i= +s
+> > already added (in which case I would expect notmuch to re-index t= +he
+> > message).
+>
+> What's the use case for these?

+

If you make a change to the message source and want it to be= + reindexed. Say, edited a draft or changed a header field. I am not asking = +that notmuch modifies the message source.

+

For _touch, if without making an actual change to the messag= +e you wish to indicate that it has been updated or synced at the current ti= +me. For instance after an reindex that did not make any actual change. Perh= +aps not strictly necessary.

+

Cheers, Gaute

+ +--001a11c35a9c71cb080503b613b8-- -- 2.26.2