From: dm-list-email-notmuch Date: Thu, 10 Apr 2014 15:31:04 +0000 (+1700) Subject: Re: [PATCH] Add configurable changed tag to messages that have been changed on disk X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=a835790d7d67f7d1a3cc3a8d972441f1925635ad;p=notmuch-archives.git Re: [PATCH] Add configurable changed tag to messages that have been changed on disk --- diff --git a/35/c9195a17fba79293dee358fdb12cea764615fc b/35/c9195a17fba79293dee358fdb12cea764615fc new file mode 100644 index 000000000..fc1f580b2 --- /dev/null +++ b/35/c9195a17fba79293dee358fdb12cea764615fc @@ -0,0 +1,134 @@ +Return-Path: + +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 2ADF3431FBD + for ; Thu, 10 Apr 2014 08:32:28 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -2.3 +X-Spam-Level: +X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id KbDkTKtL8psJ for ; + Thu, 10 Apr 2014 08:32:21 -0700 (PDT) +Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 136E8431FBC + for ; Thu, 10 Apr 2014 08:32:20 -0700 (PDT) +Received: from market.scs.stanford.edu (localhost.scs.stanford.edu + [127.0.0.1]) by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id + s3AFV4Ia005872; Thu, 10 Apr 2014 08:31:04 -0700 (PDT) +Received: (from dm@localhost) + by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id s3AFV49m008022; + Thu, 10 Apr 2014 08:31:04 -0700 (PDT) +X-Authentication-Warning: market.scs.stanford.edu: dm set sender to + return-hrv8eihpgeb84a37siqwh9vg2i@ta.scs.stanford.edu using -f +From: dm-list-email-notmuch@scs.stanford.edu +To: Gaute Hope +Subject: Re: [PATCH] Add configurable changed tag to messages that have been + changed on disk +In-Reply-To: <1397140962-sup-6514@qwerzila> +References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com> + <87wqf2gqig.fsf@ta.scs.stanford.edu> <1397140962-sup-6514@qwerzila> +Date: Thu, 10 Apr 2014 08:31:04 -0700 +Message-ID: <87wqexnqvb.fsf@ta.scs.stanford.edu> +MIME-Version: 1.0 +Content-Type: text/plain +Cc: notmuch +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +Reply-To: David Mazieres expires 2014-07-09 PDT + +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 10 Apr 2014 15:32:28 -0000 + +Gaute Hope writes: + +>> A better approach would be to add a new "modtime" xapian value that is +>> updated whenever the tags or any other terms (such as XFDIRENTRY) are +>> added to or deleted from a docid. If it's a Xapian value, rather than a +>> term, then modtime will be queriable just like date, allowing multiple +>> applications to query all docids modified since the last time they ran. +>> +>> [... snip] +> +> This could also solve it, and probably have more uses. I don't quite see +> how the opposite problem (for my use case) can be solved by this without +> using a 'localchange' tag. This is to sync tag to maildir sync, when a +> new tag has been added (by e.g. a user interaction in a client) it needs +> to be copied to the maildir, if it is not done in the same go a +> different application won't know whether the change was local or remote. +> How did you solve this? + +Why don't you just set maildir.synchronize_flags=true? When I +synchronize mail across machines, I start by concurrently running +"notmuch new" on both the local and remote machines, which picks up all +the changed maildir flags. Then I synchronize the mail and the tags +between the two maildirs. If maildir.synchronize=true, then atomically +with setting the new tags I call notmuch_message_tags_to_maildir_flags() +to sync the new tags to the maildir. + +The maildir flags question seems kind of independent of what we are +talking about, which is just having an incremental way of examining the +database. Right now, I have to scan everything to find tags that have +changed since the last synchronization event. If I had modtime (or +really it should be called "ctime", like inode change time), then I +could look at only the few messages that changed, and it would probably +shave 250msec off polling new mail for a 100,000-message maildir. + +Note you can't use the file system ctime/mtime because the file system +may have changed since the last time you ran notmuch new. + +> I would suggest using a Xapian- or Index-time which gets a tick +> everytime a modification is made to the index. + +Exactly. It could be a tick, or just the current time of day if your +clock does not go backwards. (I'd be willing to do a full scan if the +clock ever goes backwards.) The advantage of time is that you don't +have to synchronously update some counter. + +> Atomic operations could operate on the same time in case this +> distinction turns out to be useful. Perhaps something like this +> already exists in Xapian? + +I don't think it's important for atomic operations to have the same +timestamp. All that's important is that you be able to diff the +database between the last time you scanned it. + +> This way clock skew, clock resolution (lots of operations happening in +> the same second, msec or nanosec) problems won't be an issue. The crux +> will be to make sure all write-operations trigger a tick on the +> indextime. + +Clock skew is not really an issue. It takes years to amass hundreds of +thousands of email messages. So adding 5 minutes of slop is not a big +deal--you'll just scan a few messages needlessly. + +Making sure the write-operations update the time should be easy. Most +or all of the changes are probably funneled through +_notmuch_message_sync. Worst case, there are only 9 places in the +source code that make use of a Xapian:WritableDatabase, so I'm pretty +confident total changes wouldn't be much more than 50 lines of code. + +I would do it myself if there were any kind of indication that such a +change could be upstreamed. I brought this up in January, 2011, and +didn't get a huge amount of interest in the ctime idea. But I was also +a lot less focused on what I needed. Now that I have a working +distributed setup and am actually using notmuch for my mail, I have a +much better understanding of what is needed. + +David