--- /dev/null
+Return-Path:\r
+ <return-qnmtsdtvmswvfn8meiysvc9qaw@temporary-address.scs.stanford.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id D26A3431FC4\r
+ for <notmuch@notmuchmail.org>; Wed, 23 Apr 2014 15:31:58 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id PIBwrsjGS7ZV for <notmuch@notmuchmail.org>;\r
+ Wed, 23 Apr 2014 15:31:54 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id E922B431FBF\r
+ for <notmuch@notmuchmail.org>; Wed, 23 Apr 2014 15:31:53 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (localhost.scs.stanford.edu\r
+ [127.0.0.1]) by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id\r
+ s3NMVoYW028965; Wed, 23 Apr 2014 15:31:50 -0700 (PDT)\r
+Received: (from dm@localhost)\r
+ by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id s3NMVnq9018357;\r
+ Wed, 23 Apr 2014 15:31:49 -0700 (PDT)\r
+X-Authentication-Warning: market.scs.stanford.edu: dm set sender to\r
+ return-qnmtsdtvmswvfn8meiysvc9qaw@ta.scs.stanford.edu using -f\r
+From: dm-list-email-notmuch@scs.stanford.edu\r
+To: Austin Clements <amdragon@MIT.EDU>\r
+Subject: Re: [PATCH] Add configurable changed tag to messages that have been\r
+ changed on disk\r
+In-Reply-To: <20140423205920.GM25817@mit.edu>\r
+References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com>\r
+ <87wqf2gqig.fsf@ta.scs.stanford.edu> <1397140962-sup-6514@qwerzila>\r
+ <87wqexnqvb.fsf@ta.scs.stanford.edu> <1397163239-sup-5101@qwerzila>\r
+ <87d2g9ja0h.fsf@maritornes.cs.unb.ca> <1398237865-sup-624@qwerzila>\r
+ <87ioq0l8th.fsf@ta.scs.stanford.edu>\r
+ <20140423205920.GM25817@mit.edu>\r
+Date: Wed, 23 Apr 2014 15:31:49 -0700\r
+Message-ID: <87iopz4qzu.fsf@ta.scs.stanford.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+Reply-To: David Mazieres expires 2014-07-22 PDT\r
+ <mazieres-x4xr8vne3sssqcz9ra4ggm9ris@temporary-address.scs.stanford.edu>\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 23 Apr 2014 22:31:59 -0000\r
+\r
+Austin Clements <amdragon@MIT.EDU> writes:\r
+\r
+>> A middle ground might be to use the maximum of two values: 1) the\r
+>> time-of-day at which notmuch started executing, and 2) the highest ctime\r
+>> in the database plus 100 microseconds (leaving plenty of slop to store\r
+>> timestamps as IEEE doubles with 52 significant bits). Since the values\r
+>> will be Btree-indexed, computing the max plus one will be cheap.\r
+>\r
+> This makes me curious if you've considered how to fit this in to\r
+> Xapian. The Xapian query syntax supports range queries over document\r
+> "values", but within the Xapian B-tree, values are stored in docid\r
+> order, not value order, so Xapian's range query operator is actually a\r
+> full scan in implementation. I assume it does this so it doesn't have\r
+> to store both forward and inverse indexes of values. (I spent some\r
+> time figuring out the layout of the Xapian database and have fairly\r
+> detailed notes if anyone's curious.)\r
+\r
+Aside from finding the previous max time, everything else should work\r
+identically to the date query operator and NOTMUCH_VALUE_TIMESTAMP.\r
+\r
+Though I believe you, I'm a little surprised the values aren't indexed.\r
+An alternative design might use terms like XCTIMExxxxxxxxxxxxxxxx where\r
+the x's are hex digits. But this seems a bit clunky and not using\r
+Xapian the way it is indented to be used.\r
+\r
+When I do a query with a giant result set ordered by date (notmuch\r
+search --sort=oldest-first "*"), the first few results come back pretty\r
+quickly, so I guess the full database scan is not an issue, at least for\r
+~10^5 messages.\r
+\r
+> This is still reasonably fast in practice because it's a sequential\r
+> scan and only requires a few bytes per message, but it's probably not\r
+> what you'd expect. That said, Xapian does track per-value statistics\r
+> that would suffice for the particular problem of monotonic time stamps\r
+> (e.g., Database::get_value_upper_bound).\r
+\r
+Oh, well in that case there is no issue. That max is the only statistic\r
+we need. Everything that requires a full database scan, like get me all\r
+messages whose properties have changed since time X, is something that\r
+you can't do at all right now. And in fact I'm already scanning all\r
+100,000 message IDs AND diffing the results against a separate sqlite\r
+database to detect changes in only 0.09 seconds (Linux) or 1.2 seconds\r
+(32-bit OpenBSD). This will only make that faster, and additionally\r
+allow other people to do what I'm doing without writing a bunch of C++\r
+code.\r
+\r
+> In principle it would be possible to use user metadata or even\r
+> document terms to support true B-tree range scans by ctime order, but\r
+> I don't think it's possible to express queries over this using\r
+> Xapian's query parser. I've written about 90% of a (new) custom query\r
+> parser for Notmuch that would enable this, but little things like my\r
+> looming thesis deadline have interfered with me finishing it.\r
+\r
+Yeah, I've been avoiding the query parser and just scanning terms and\r
+postlists directly. Since the lack of ctime forced me to scan the whole\r
+database anyway, I found it much faster to scan each tag's posting list\r
+and dump the results into sqlite than to extract tag terms on a\r
+per-document basis the way notmuch dump does.\r
+\r
+>> Incidentally, if you are really this paranoid about time stamps, it\r
+>> should bother you that notmuch's directory timestamps only have one\r
+>> second granularity.\r
+>\r
+> This is historical (and, I agree, unfortunate). But nobody's\r
+> complained, so it hasn't been worth changing the libnotmuch interface\r
+> to support sub-second directory mtimes. However, notmuch new does\r
+> correctly handle deliveries in the same second it runs. If the\r
+> wall-clock time when it starts is the same as the on-disk directory\r
+> mtime, it skips updating the in-database directory mtime at the end.\r
+> Hence, on the next run, it will still consider the directory\r
+> out-of-date. It's a bit of a hack, but it's a hack that would be\r
+> necessary for supporting older file systems even if we did support\r
+> sub-second timestamps.\r
+\r
+Yeah, is kind of a problem me. I currently scan the XFDIRENTRY terms\r
+belonging to a directory only if the directory's notmuch mtime has\r
+changed since the last time I examined Xapian's state. I used to scan\r
+the actual directories, which was fine, but not so useful because I\r
+don't actually want to deal with messages that notmuch has not yet\r
+indexed. Conversely, if a directory has not changed since the last time\r
+muchsync ran, but notmuch's idea of the directory has changed (because\r
+someone ran notmuch new), then I do care about scanning for new/deleted\r
+XFDIRENTRY terms.\r
+\r
+But couldn't notmuch fix the sub-second problem in a fully backwards\r
+compatible way? After all, the database is already storing these mtimes\r
+as doubles. Even for OSes that don't support st_mtim, notmuch could\r
+just add 0.00001 seconds to the previous timestamp of a modified\r
+directory.\r
+\r
+David\r
projects / notmuch-archives.git / commitdiff