--- /dev/null
+Return-Path:\r
+ <return-yjfumptdmm9v8zs6su3fi692c6@temporary-address.scs.stanford.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id D6677421176\r
+ for <notmuch@notmuchmail.org>; Fri, 11 Apr 2014 09:05:00 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id oHihRs1iJvsa for <notmuch@notmuchmail.org>;\r
+ Fri, 11 Apr 2014 09:04:56 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 937E6421173\r
+ for <notmuch@notmuchmail.org>; Fri, 11 Apr 2014 09:04:56 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (localhost.scs.stanford.edu\r
+ [127.0.0.1]) by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id\r
+ s3BG3c97012339; Fri, 11 Apr 2014 09:03:38 -0700 (PDT)\r
+Received: (from dm@localhost)\r
+ by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id s3BG3cFO000873;\r
+ Fri, 11 Apr 2014 09:03:38 -0700 (PDT)\r
+X-Authentication-Warning: market.scs.stanford.edu: dm set sender to\r
+ return-yjfumptdmm9v8zs6su3fi692c6@ta.scs.stanford.edu using -f\r
+From: dm-list-email-notmuch@scs.stanford.edu\r
+To: David Bremner <david@tethera.net>, Gaute Hope <eg@gaute.vetsj.com>\r
+Subject: Re: [PATCH] Add configurable changed tag to messages that have been\r
+ changed on disk\r
+In-Reply-To: <87k3aw5dj5.fsf@zancas.localnet>\r
+References: <1396800683-9164-1-git-send-email-eg@gaute.vetsj.com>\r
+ <87wqf2gqig.fsf@ta.scs.stanford.edu> <1397140962-sup-6514@qwerzila>\r
+ <87wqexnqvb.fsf@ta.scs.stanford.edu>\r
+ <87k3aw5dj5.fsf@zancas.localnet>\r
+Date: Fri, 11 Apr 2014 09:03:38 -0700\r
+Message-ID: <877g6v3lb9.fsf@ta.scs.stanford.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+Cc: notmuch <notmuch@notmuchmail.org>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+Reply-To: David Mazieres expires 2014-07-10 PDT\r
+ <mazieres-7je64uqare55j4eig77r2axtva@temporary-address.scs.stanford.edu>\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 11 Apr 2014 16:05:01 -0000\r
+\r
+David Bremner <david@tethera.net> writes:\r
+\r
+>> Exactly. It could be a tick, or just the current time of day if your\r
+>> clock does not go backwards. (I'd be willing to do a full scan if the\r
+>> clock ever goes backwards.) The advantage of time is that you don't\r
+>> have to synchronously update some counter.\r
+>\r
+> I think I'd lean towards global time so that one could use it to resolve\r
+> conflicts between changes to multiple copies of the database.\r
+\r
+I, too, would prefer to use time. However, I'm doubtful it would help\r
+resolve conflicts. On the plus side, I'm not sure it is even needed to\r
+resolve conflicts. My mail synchronizer has an algorithm for resolving\r
+conflicts that always works without human intervention and in my limited\r
+experience does exactly what I want:\r
+\r
+ * If there's a conflict between two replicas, ensure that each\r
+ maildir ends up with the maximum number of the number copies of the\r
+ message in each of the two databases being reconciled. [Example:\r
+ If replica A deletes a message and replica B moves it from folder\r
+ INBOX to folder SPAM, you end up with a copy in spam. If replica A\r
+ moves a message to folder IMPORTANT and replica B moves it to SPAM,\r
+ then you get two hard links to the same file, one in IMPORTANT and\r
+ one in SPAM.]\r
+\r
+ * If there's a conflict and two replicas have different tags on the\r
+ same message, then the tags in notmuch's new.tags directive get\r
+ logically ANDed, while all other tags get logically ORed.\r
+\r
+Granted, I've only been using this system for a week. On the other\r
+hand, all I was doing was starting to test something I had written, yet\r
+it ended up being so much better than my old system that I couldn't go\r
+back and ended up using my system in production far earlier than\r
+anticipated...\r
+\r
+>> Making sure the write-operations update the time should be easy. Most\r
+>> or all of the changes are probably funneled through\r
+>> _notmuch_message_sync. Worst case, there are only 9 places in the\r
+>> source code that make use of a Xapian:WritableDatabase, so I'm pretty\r
+>> confident total changes wouldn't be much more than 50 lines of code.\r
+>\r
+> Maybe. Don't forget upgrading the database, updating the test suite, and\r
+> presumably some changes to the CLI so the new mtime can actually be\r
+> used. Not to be discouraging ;).\r
+\r
+The CLI is trivial. We'll just add another search keyword ctime\r
+analogous to date.\r
+\r
+As far as updating the test suite, etc., it's almost certain that the\r
+core notmuch developers would be unsatisfied with whatever I've done,\r
+since the code base is very clean and has a very uniform style. So when\r
+I say I'd want some "indication that such a change could be upstreamed,"\r
+I mean more specifically that someone would be willing to shepherd the\r
+process of getting the code into shape.\r
+\r
+> In the ensuing time, nothing better has developed for tag\r
+> synchronization (my pet use case) so maybe it's time to pursue this\r
+> again.\r
+\r
+I do have something pretty good for tag synchronization. It requires a\r
+full database scan each time to detect changes, but I've heavily\r
+optimized it to be very fast by skipping over the notmuch library and\r
+directly scanning the underlying Xapian Btrees. Currently my bottleneck\r
+is indexing messages (e.g., running notmuch new or calling\r
+notmuch_database_add_message), which are painfully slow on 32-bit\r
+machines. (Unfortunately my mail server is a 32-bit machine.)\r
+\r
+To give you an idea, on a 32 bit machine, if I get a handful of new mail\r
+(e.g., 6 messages), running "notmuch new" takes 19 seconds, while\r
+scanning the database to check for renames and changed tags adds another\r
+1.4 seconds. On a 64-bit machine, "notmuch new" might take 1 second,\r
+while scanning the database adds 350 msec.\r
+\r
+So full database scan's might not be the end of the world. The biggest\r
+performance bottleneck at this point is notmuch's painful indexing\r
+performance. It kills me that it takes 10 minutes to index 100,000 mail\r
+messages on a 16-core machine with 48 GiB of RAM. But the library is\r
+non-reentrant and allocates thread IDs in such a way that it's hard to\r
+create parallel databases and later merge them. Basically I can't\r
+figure out how to make productive use of more than one CPU core even\r
+when synchronizing across 1GB Ethernet!\r
+\r
+It's pretty beta, but my intention is to open-source my code, so glad\r
+for beta testers if you are interested in testing tag synchronization.\r
+\r
+> It would be good to have some preliminary idea about the time\r
+> and space costs of adding document mtimes. I guess database bloat\r
+> should not be too bad, since it's only 64bits (?) per mail message.\r
+\r
+Plus a Btree to index it, so figure at least 24 bytes per message.\r
+Another issue is that values are always brought into memory with a\r
+document, so it will consume more RAM. But yeah, I don't think it\r
+should be that bad.\r
+\r
+David\r
projects / notmuch-archives.git / commitdiff