--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 09C44431FBD\r
+ for <notmuch@notmuchmail.org>; Wed, 23 Apr 2014 13:59:37 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id SXgPjVp1H138 for <notmuch@notmuchmail.org>;\r
+ Wed, 23 Apr 2014 13:59:29 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-5.mit.edu (dmz-mailsec-scanner-5.mit.edu\r
+ [18.7.68.34])\r
+ by olra.theworths.org (Postfix) with ESMTP id 4AFA9431FAE\r
+ for <notmuch@notmuchmail.org>; Wed, 23 Apr 2014 13:59:29 -0700 (PDT)\r
+X-AuditID: 12074422-f79186d00000135a-cf-535829b022e6\r
+Received: from mailhub-auth-2.mit.edu ( [18.7.62.36])\r
+ (using TLS with cipher AES256-SHA (256/256 bits))\r
+ (Client did not present a certificate)\r
+ by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 39.09.04954.0B928535; Wed, 23 Apr 2014 16:59:28 -0400 (EDT)\r
+Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
+ by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id s3NKxQ0j020016; \r
+ Wed, 23 Apr 2014 16:59:27 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s3NKxMNb008365\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+ Wed, 23 Apr 2014 16:59:24 -0400\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+ (envelope-from <amdragon@mit.edu>)\r
+ id 1Wd4GI-0004jG-1m; Wed, 23 Apr 2014 16:59:22 -0400\r
+Date: Wed, 23 Apr 2014 16:59:20 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: David Mazieres expires 2014-07-22 PDT\r
+ <mazieres-eqbxrjbqrn7vs5xdx45e9hkucs@temporary-address.scs.stanford.edu>\r
+Subject: Re: [PATCH] Add configurable changed tag to messages that have been\r
+ changed on disk\r
+Message-ID: <20140423205920.GM25817@mit.edu>\r
+References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com>\r
+ <87wqf2gqig.fsf@ta.scs.stanford.edu> <1397140962-sup-6514@qwerzila>\r
+ <87wqexnqvb.fsf@ta.scs.stanford.edu> <1397163239-sup-5101@qwerzila>\r
+ <87d2g9ja0h.fsf@maritornes.cs.unb.ca> <1398237865-sup-624@qwerzila>\r
+ <87ioq0l8th.fsf@ta.scs.stanford.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <87ioq0l8th.fsf@ta.scs.stanford.edu>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFprOKsWRmVeSWpSXmKPExsUixG6nortBMyLY4OFaFYsbrd2MFk2fL7Fa\r
+ HJ/+hc3i+s2ZzA4sHj/+NbN5PFt1i9nj0t9tTB5bDr1nDmCJ4rJJSc3JLEst0rdL4Mo4OMWq\r
+ oEe2YsbtE6wNjNvFuxg5OSQETCSaFs1kg7DFJC7cWw9kc3EICcxmklh7dw8jhLORUWLhyVfs\r
+ EM5pJol36/exQDhLGCV+Hf7KAtLPIqAq8XPZT7BZbAIaEtv2L2cEsUUEiiSur/wPFmcGsk/v\r
+ 3A1WLywQJzHt8g4mEJtXQEfiQvdEZhBbSOAAk8TOP3wQcUGJkzOfsED0aknc+PcSqJ4DyJaW\r
+ WP6PAyTMKWAosafrPTuILSqgIjHl5Da2CYxCs5B0z0LSPQuhewEj8ypG2ZTcKt3cxMyc4tRk\r
+ 3eLkxLy81CJdU73czBK91JTSTYzg8HdR2sH486DSIUYBDkYlHt4DF8KDhVgTy4orcw8xSnIw\r
+ KYnyqilGBAvxJeWnVGYkFmfEF5XmpBYfYpTgYFYS4c3SAMrxpiRWVqUW5cOkpDlYlMR531pb\r
+ BQsJpCeWpGanphakFsFkZTg4lCR4z4E0ChalpqdWpGXmlCCkmTg4QYbzAA2fBja8uCAxtzgz\r
+ HSJ/ilFRSpx3mzpQQgAkkVGaB9cLS0+vGMWBXhHmLQdp5wGmNrjuV0CDmYAGF0wIBxlckoiQ\r
+ kmpgnF/YalHz60r/hKxd/IFSQdeuPDnm+K98SrmW9aPN0lGuSqyp1X4ZR/lOGD7dOeFfwcvV\r
+ B6/qhuct3uK9TenwhMjFBTPbr0v8qV/t1ZbWdFpY/zTbvbevH3FLvWf+LLlc7vvUf+vL7S/M\r
+ W3Xhcr7blGX3D9UrVyXov3+jaiW1/na0B+ORzoWs3UosxRmJhlrMRcWJAJ0nCSsqAwAA\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 23 Apr 2014 20:59:37 -0000\r
+\r
+Hi Dave!\r
+\r
+Quoth David Mazieres on Apr 23 at 2:00 am:\r
+> Gaute Hope <eg@gaute.vetsj.com> writes:\r
+> \r
+> > A db-tick or a _good_ ctime solution can as far as I can see solve both\r
+> > David M's (correct me if I am wrong) and my purposes, as well as\r
+> > probably have more use cases in the future. It would even be an\r
+> > interesting direct search: show me everything that changed lately,\r
+> > sorted.\r
+> \r
+> I could live with a db-tick scheme. I would prefer a ctime scheme,\r
+> since then I can answer questions such as "what has changed in the last\r
+> five minutes"? I mean all kinds of other stuff starts to break if your\r
+> clock goes backwards on a mail server machine, not the least of which is\r
+> that incremental backups will fail silently, so you risk losing your\r
+> mail.\r
+> \r
+> A middle ground might be to use the maximum of two values: 1) the\r
+> time-of-day at which notmuch started executing, and 2) the highest ctime\r
+> in the database plus 100 microseconds (leaving plenty of slop to store\r
+> timestamps as IEEE doubles with 52 significant bits). Since the values\r
+> will be Btree-indexed, computing the max plus one will be cheap.\r
+\r
+This makes me curious if you've considered how to fit this in to\r
+Xapian. The Xapian query syntax supports range queries over document\r
+"values", but within the Xapian B-tree, values are stored in docid\r
+order, not value order, so Xapian's range query operator is actually a\r
+full scan in implementation. I assume it does this so it doesn't have\r
+to store both forward and inverse indexes of values. (I spent some\r
+time figuring out the layout of the Xapian database and have fairly\r
+detailed notes if anyone's curious.)\r
+\r
+This is still reasonably fast in practice because it's a sequential\r
+scan and only requires a few bytes per message, but it's probably not\r
+what you'd expect. That said, Xapian does track per-value statistics\r
+that would suffice for the particular problem of monotonic time stamps\r
+(e.g., Database::get_value_upper_bound).\r
+\r
+In principle it would be possible to use user metadata or even\r
+document terms to support true B-tree range scans by ctime order, but\r
+I don't think it's possible to express queries over this using\r
+Xapian's query parser. I've written about 90% of a (new) custom query\r
+parser for Notmuch that would enable this, but little things like my\r
+looming thesis deadline have interfered with me finishing it.\r
+\r
+> Incidentally, if you are really this paranoid about time stamps, it\r
+> should bother you that notmuch's directory timestamps only have one\r
+> second granularity. It's not that hard to get a new message delivered\r
+> in the same second that notmuch new finished running. In my\r
+> synchronizer, I convert st_mtim (a struct timespec) into a double and\r
+> keep that plus size in the database to decide if I need to re-hash\r
+> files. But for directories, I'm stuck with NOTMUCH_VALUE_TIMESTAMP,\r
+> which are quantized to the second. (Ironically, I think\r
+> Xapian::sortable_serialize converts time_ts to doubles anyway, so\r
+> avoiding st_mtim is not really helping performance.)\r
+\r
+This is historical (and, I agree, unfortunate). But nobody's\r
+complained, so it hasn't been worth changing the libnotmuch interface\r
+to support sub-second directory mtimes. However, notmuch new does\r
+correctly handle deliveries in the same second it runs. If the\r
+wall-clock time when it starts is the same as the on-disk directory\r
+mtime, it skips updating the in-database directory mtime at the end.\r
+Hence, on the next run, it will still consider the directory\r
+out-of-date. It's a bit of a hack, but it's a hack that would be\r
+necessary for supporting older file systems even if we did support\r
+sub-second timestamps.\r
projects / notmuch-archives.git / commitdiff