From 85789dc6ea14f916e56c7bb391f90d0a3fffa61c Mon Sep 17 00:00:00 2001 From: Austin Clements Date: Tue, 23 Sep 2014 11:40:09 +2000 Subject: [PATCH] Re: [PATCH] Add configurable changed tag to messages that have been changed on disk --- 24/faa14dcc27e2f99383e505cd92d91b01bcb3ff | 163 ++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 24/faa14dcc27e2f99383e505cd92d91b01bcb3ff diff --git a/24/faa14dcc27e2f99383e505cd92d91b01bcb3ff b/24/faa14dcc27e2f99383e505cd92d91b01bcb3ff new file mode 100644 index 000000000..b97eb0b8b --- /dev/null +++ b/24/faa14dcc27e2f99383e505cd92d91b01bcb3ff @@ -0,0 +1,163 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id E8D37431FC4 + for ; Mon, 22 Sep 2014 08:40:40 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -2.3 +X-Spam-Level: +X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id AcXiMF6aUGtd for ; + Mon, 22 Sep 2014 08:40:34 -0700 (PDT) +Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu + [18.9.25.15]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id D7C12431FB6 + for ; Mon, 22 Sep 2014 08:40:33 -0700 (PDT) +X-AuditID: 1209190f-f79aa6d000005b45-cc-542042f1fcc5 +Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) + (using TLS with cipher AES256-SHA (256/256 bits)) + (Client did not present a certificate) + by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP + id 42.96.23365.1F240245; Mon, 22 Sep 2014 11:40:33 -0400 (EDT) +Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) + by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id s8MFeWQn028356; + Mon, 22 Sep 2014 11:40:32 -0400 +Received: from drake.dyndns.org + (HSI-KBW-109-192-025-091.hsi6.kabel-badenwuerttemberg.de + [109.192.25.91]) (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s8MFeDNc030355 + (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); + Mon, 22 Sep 2014 11:40:29 -0400 +Received: from amthrax by drake.dyndns.org with local (Exim 4.84) + (envelope-from ) + id 1XW5ik-0005OB-KO; Mon, 22 Sep 2014 11:40:10 -0400 +From: Austin Clements +To: Gaute Hope , notmuch@notmuchmail.org +Subject: Re: [PATCH] Add configurable changed tag to messages that have + been changed on disk +In-Reply-To: <1411386960-astroid-2-k1e726ut3f-2518@strange> +References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com> + <87fviiiuzn.fsf@maritornes.cs.unb.ca> + + <20140801185505.GS13893@mit.edu> + <1407313144-astroid-0-vyhth1tcrd-3835@strange> + <1411386960-astroid-2-k1e726ut3f-2518@strange> +User-Agent: Notmuch/0.18.1+86~gef5e66a (http://notmuchmail.org) Emacs/24.3.1 + (x86_64-pc-linux-gnu) +Date: Mon, 22 Sep 2014 11:40:09 -0400 +Message-ID: <87k34vackm.fsf@drake.dyndns.org> +MIME-Version: 1.0 +Content-Type: text/plain +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFnrPIsWRmVeSWpSXmKPExsUixG6novvRSSHEYOFCLoumz5dYLa7fnMns + wOTx418zm8ezVbeYA5iiuGxSUnMyy1KL9O0SuDI+f+xmKtgvW/G9t5exgfGfeBcjJ4eEgInE + ih3HmSBsMYkL99azdTFycQgJzGaS+PjtPCOEs5FR4uDpDVCZ+0wSk7ouskM4cxklJqycxw7S + zyagL7Fi7SRWEFtEwEbi1Pr9YLawQIzEvQOvGEFsTgFriXs3HjJDNK9mkjj6eDNYQlQgSWLx + ojnMIDaLgKrEhxPLweK8AroSXdcOMEPYghInZz5hAbGZBSQkDr54wTyBUWAWktQsJKkFjEyr + GGVTcqt0cxMzc4pTk3WLkxPz8lKLdE30cjNL9FJTSjcxgsKSU5J/B+O3g0qHGAU4GJV4eBc0 + yYcIsSaWFVfmHmKU5GBSEuW9b6oQIsSXlJ9SmZFYnBFfVJqTWnyIUYKDWUmEN0gaKMebklhZ + lVqUD5OS5mBREufd9IMvREggPbEkNTs1tSC1CCYrw8GhJMFrCYw/IcGi1PTUirTMnBKENBMH + J8hwHqDhXiA1vMUFibnFmekQ+VOMuhzrOr/1Mwmx5OXnpUqJ8yqBFAmAFGWU5sHNgaWTV4zi + QG8J8xqBVPEAUxHcpFdAS5iAltw/Lg+ypCQRISXVwDi/OfsG46ufgpMTt3jIWOYfm1rD8DDu + aHT3iy9ed29fMOitVemokf2W0FN626ktM996i9LzD/nuux/wf5m3+Mn+tV9XJlXP6juSELVW + n1/nzf3W8Puxb63u8Lasv2VnYFaj0KAu85pZ8/jPm2x6wTOvrfVqOSJlq3pM7rP92sCjrhw3 + +ebfaldiKc5INNRiLipOBABOaxnpAgMAAA== +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 22 Sep 2014 15:40:41 -0000 + +On Mon, 22 Sep 2014, Gaute Hope wrote: +> Excerpts from Gaute Hope's message of August 6, 2014 10:29: +>> Austin Clements wrote on Fri, 01 Aug 2014 14:55:05 -0400: +>>> I have a prototype implementation of message modification times on my +>>> lastmod-v1 branch at +>>> +>>> https://github.com/aclements/notmuch/tree/lastmod-v1 +>>> +>>> It builds on my database features series that's currently awaiting +>>> review [1]. +>>> +>>> The series uses a monotonic revision number, rather than wall-clock +>>> time, for reasons related to Xapian's concurrent control and detailed +>>> in the main commit's commit message. The implementation isn't quite +>>> useful from the CLI yet because I haven't added any way to query the +>>> database's current revision number. (I'm still thinking about how I +>>> want to do this, since search/show don't have a good way to deliver +>>> "additional" information right now. I might just add the last +>>> modification for each individual message/max of all messages in a +>>> thread, similar to what Thomas Jost's patch did long ago.) +>>> +>>> [1] id:1406859003-11561-1-git-send-email-amdragon@mit.edu +> +>> this should allow me to do what I wish to accomplish. The message +>> deletion is still a problem though, I can see two options at the moment: +> +> Hi list, +> +> While exploring the possibility of syncing maildir/X-keywords with tags +> I had some thoughts about lastmod and message modification: +> +> As briefly discussed on #notmuch, I noticed that it seems that 'notmuch +> new' does not detect that a message source has been changed, unless the +> file is also re-named. +> +> This means that for instance if the X-Keywords fields have been updated +> in a message (from GMail with offlineimap, synclabels = yes) the lastmod +> field will remain unchanged, and a source modification will be +> undetectable to a client program using this value. +> +> Would it not make sense that if a message has a more recent mtime than +> at index time it is re-indexed? + +This has the potential to make notmuch new substantially more expensive. +Currently, if there are no changes, it only has to stat each directory +in your maildir (in fact, some restructuring of new would let us +eliminate almost all database access during a no-op notmuch new as +well). Checking for changes to individual messages would require +stat'ing every single message file as well as accessing the database to +check the paths and mtimes of every message, increasing the number of +stat calls and disk accesses by several orders of magnitude. + +It may be that this is fast enough that it's okay, but it would be good +to gather some evidence first. That includes hot and cold caches, and +maildir over NFS. + +With respect to X-Keywords specifically, note that it's a fairly basic +design decision that notmuch never modifies message files. This gives +us strong robustness guarantees we would be loathe to part with. + +It has puzzled me ever since offlineimap added X-Keywords why they +didn't just translate these keywords into folders and create hard links +of message files. Anything could interact smoothly with that. + +> Also, for the lastmod branch I would wish for a notmuch_message_touch() +> method where the lastmod time is updated to the last. As well as a +> notmuch_database_reindex_message () - possibly defined/documented +> behaviour for notmuch_database_add_message () when the filename is +> already added (in which case I would expect notmuch to re-index the +> message). + +What's the use case for these? + +> Doing notmuch_database_remove_message followed by _add_message could +> risk deleting the entry if this file is the only on-disk-representation. +> +> Cheers, Gaute -- 2.26.2