--- /dev/null
+Return-Path:\r
+ <return-axb7s9swz7zfq6nbxfhh8ym7p2@temporary-address.scs.stanford.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 15EA4431FBD\r
+ for <notmuch@notmuchmail.org>; Sun, 13 Apr 2014 12:52:59 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id 3hkX2x0OY0lL for <notmuch@notmuchmail.org>;\r
+ Sun, 13 Apr 2014 12:52:50 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 27553431FBC\r
+ for <notmuch@notmuchmail.org>; Sun, 13 Apr 2014 12:52:50 -0700 (PDT)\r
+Received: from market.scs.stanford.edu (localhost.scs.stanford.edu\r
+ [127.0.0.1]) by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id\r
+ s3DJqmB5016262; Sun, 13 Apr 2014 12:52:48 -0700 (PDT)\r
+Received: (from dm@localhost)\r
+ by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id s3DJqlUW024376;\r
+ Sun, 13 Apr 2014 12:52:47 -0700 (PDT)\r
+X-Authentication-Warning: market.scs.stanford.edu: dm set sender to\r
+ return-axb7s9swz7zfq6nbxfhh8ym7p2@ta.scs.stanford.edu using -f\r
+From: dm-list-email-notmuch@scs.stanford.edu\r
+To: Tilmann Singer <tils@tils.net>, Brian Sniffen <bsniffen@akamai.com>,\r
+ notmuch@notmuchmail.org\r
+Subject: Re: Synchronization success stories?\r
+In-Reply-To: <8738hhvygu.fsf@tils.net>\r
+References: <m2r455f270.fsf@usma1mc-0csx92.kendall.corp.akamai.com>\r
+ <87ppklwin6.fsf@tils.net> <87siphl0cl.fsf@ta.scs.stanford.edu>\r
+ <8738hhvygu.fsf@tils.net>\r
+Date: Sun, 13 Apr 2014 12:52:47 -0700\r
+Message-ID: <87ppklknw0.fsf@ta.scs.stanford.edu>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+Reply-To: David Mazieres expires 2014-07-12 PDT\r
+ <mazieres-z3itifemqmn227pvca6hpue6x6@temporary-address.scs.stanford.edu>\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 13 Apr 2014 19:52:59 -0000\r
+\r
+Tilmann Singer <tils@tils.net> writes:\r
+\r
+> David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:\r
+>> What happens if you get a message that's been stuck in a queue for a few\r
+>> days and has an old Date: header?\r
+>\r
+> It would be missed. I have set the timespan to look backwards for new\r
+> mail to one month to be a bit safer against the stuck-in-queue cases,\r
+> but mails with older Date: headers would definitely get missed.\r
+>\r
+> The current output of notmuch count "*" is the same on both the client\r
+> and the server, so it seems I didn't run into this problem yet (maybe I\r
+> was just lucky).\r
+\r
+I've been playing around with reorganizing my maildir, and found a\r
+couple of messages (on mailing lists) with clearly invalid dates years\r
+in the past. But checking with notmuch count is a good idea. Then you\r
+can always fall back to the slow path in the unlikely event that your\r
+counts don't match up. Well, except that A) count is just unique\r
+message-IDs, not messages, and B) when synchronizing in both directions\r
+you could still miss something. You have to assume that the invalid\r
+dates are only ever going to occur at one end of a synchronization\r
+event.\r
+\r
+>> Or if you get new messages that have\r
+>> the same Message-ID as old ones?\r
+>\r
+> Is that even possible? I thought that notmuch guarantees the uniqueness\r
+> of indexed message ids. The only reference I could find without trying\r
+> to read the code was this thread id:87mwyz3s9d.fsf@star.eba from 2012,\r
+> which supports the assumption.\r
+\r
+Sadly, yes it is quite possible, and even opens up a slight security\r
+issue. Suppose I know you are on a mailing list, and some message\r
+appears on that mailing list that I don't want you to see. I can send\r
+you an innocuous-looking message that just happens to have the same\r
+message-id, and you may never see the original mailing list message.\r
+Even better, depending on how your spam filtering is setup, if I include\r
+the GTUBE string in my message you may never see mine or the original.\r
+\r
+That's why with muchsync, I replicate actual mail messages, rather than\r
+message-IDs. Then you can always periodically check for message-IDs\r
+that appear in more than one file. (In fact, thought I haven't\r
+published an interface for this, the SQL database kept my muchsync makes\r
+it trivial to check for this and detect certain attacks.)\r
+\r
+I understand why notmuch went with message IDs. For instance you have\r
+sent this reply both directly to me and to a mailing list I am\r
+subscribed to. So I will get two slightly different copies of the\r
+message (one will have the standard notmuch mailing list signature, the\r
+other won't). And this way once I've marked it read, the message will\r
+be read even once the second copy comes in. But personally I'd rather\r
+see the occasional duplicate message than risk not seeing messages. In\r
+particular, if the goal is to see fewer unread messages, some sort of\r
+feature that pro-actively skips all future messages in a thread or\r
+subthread would be more useful...\r
+\r
+> Here is how long they take (on a machine with an SSD, which certainly\r
+> helps):\r
+>\r
+> $ time notmuch dump --format=batch-tag | sort > /tmp/notmuch.dump\r
+> real 0m3.643s\r
+> user 0m3.593s\r
+> sys 0m0.140s\r
+> $ time notmuch restore < /tmp/notmuch.dump\r
+> real 0m3.719s\r
+> user 0m3.357s\r
+> sys 0m0.357s\r
+> $ notmuch count \r
+> 117118\r
+\r
+That's crazy. I'm jealous. Then again, this is how fast muchsync runs\r
+(including a full database scan to detect changed messages and tags)\r
+when there is no new mail:\r
+\r
+$ time ./muchsync -v\r
+[notmuch] No new mail.\r
+synchronizing muchsync database with Xapian... 0.038506 (+0.038506)\r
+starting scan of Xapian database... 0.039069 (+0.000563)\r
+opened Xapian... 0.040851 (+0.001782)\r
+scanned message IDs... 0.137647 (+0.096796)\r
+scanned tags... 0.170404 (+0.032757)\r
+scanned directories in xapian... 0.172100 (+0.001696)\r
+scanned filenames in xapian... 0.172376 (+0.000276)\r
+adjusted link counts... 0.199461 (+0.027085)\r
+finished synchronizing muchsync database with Xapian... 0.212965 (+0.013505)\r
+\r
+real 0m0.220s\r
+user 0m0.173s\r
+sys 0m0.023s\r
+\r
+David\r
projects / notmuch-archives.git / commitdiff