Return-Path: X-Original-To: notmuch@notmuchmail.org Delivered-To: notmuch@notmuchmail.org Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 01512431FDA for ; Fri, 3 Oct 2014 07:19:32 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -2.3 X-Spam-Level: X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ogxsEZTJupP7 for ; Fri, 3 Oct 2014 07:19:25 -0700 (PDT) Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu [18.9.25.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 8D5C8431FC0 for ; Fri, 3 Oct 2014 07:19:25 -0700 (PDT) X-AuditID: 1209190f-f79aa6d000005b45-a6-542eb06b895a Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id 3B.24.23365.B60BE245; Fri, 3 Oct 2014 10:19:23 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id s93EJNx7018546 for ; Fri, 3 Oct 2014 10:19:23 -0400 Received: from drake.dyndns.org (216-15-114-40.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com [216.15.114.40]) (authenticated bits=0) (User authenticated as amdragon@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s93EJLC0023304 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Fri, 3 Oct 2014 10:19:23 -0400 Received: from amthrax by drake.dyndns.org with local (Exim 4.84) (envelope-from ) id 1Xa3hY-0002JV-Tt for notmuch@notmuchmail.org; Fri, 03 Oct 2014 10:19:20 -0400 From: Austin Clements To: notmuch@notmuchmail.org Subject: [PATCH 00/11] Add ghost messages and fix thread linking Date: Fri, 3 Oct 2014 10:19:07 -0400 Message-Id: <1412345958-8278-1-git-send-email-aclements@csail.mit.edu> X-Mailer: git-send-email 2.1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprNIsWRmVeSWpSXmKPExsUixCmqrZu9QS/E4GKbjcX1mzOZHRg9nq26 xRzAGMVlk5Kak1mWWqRvl8CV0X3yMGvBGvaKB4t3sTQwdrJ1MXJySAiYSNx9+Z8VwhaTuHBv PVCci0NIYDaTxN2Od8wgCSGBc4wSF5+FQiR+Mkl0nfvDAuEcYpT4//IgWBWbgL7EirWTwEaJ CEhL7Lw7G8wWFnCQ2HD7MQuIzSKgKrF91VGwel4BN4l7f++zQ6yWk9iw+z/jBEaeBYwMqxhl U3KrdHMTM3OKU5N1i5MT8/JSi3RN9HIzS/RSU0o3MYL9Lcm/g/HbQaVDjAIcjEo8vB9u6IYI sSaWFVfmHmKU5GBSEuWtXKwXIsSXlJ9SmZFYnBFfVJqTWgx0MQezkgjvgpVAOd6UxMqq1KJ8 mJQ0B4uSOO+mH3whQgLpiSWp2ampBalFMFkZDg4lCd729UCNgkWp6akVaZk5JQhpJg5OkOE8 QMPBaniLCxJzizPTIfKnGHU5Wpre9jIJseTl56VKifM+XAdUJABSlFGaBzcHFqevGMWB3hLm ZQJGrRAPMMbhJr0CWsIEtOSdvS7IkpJEhJRUA+OECMHH+0VLw0WdlPQdOrh/acbES515n8/P /P7xepnpnRVvHNolZm9xFu/Ta9CwWK1x9ebmRKf/2rtcNmw5WPj1YsSxvrVOfV2bZkYccXec LH6aZ1beWqew782Gpa5TWOY9z2NjqTT0Z5lzUuuQgs0nwYlKeT8/ePrcaFNsfJDY6bhiztob /5RYijMSDbWYi4oTAeQvl6SuAgAA X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 14:19:32 -0000 This series modifies our database representation of messages that have been referenced by other messages, but for which we don't have the message itself. Currently, we store this information as Xapian metadata, but this has several downsides for performance and complexity and results in hard-to-fix thread linking bugs. This patch series implements "ghost messages", which replace this Xapian metadata with Xapian documents that look and act very much like regular message documents, but simply have no content. This simplifies and speeds up our thread linking algorithm and fixes the currently broken thread linking test. Ghost messages also open up interesting future possibilities, such as "pre-seeding" tags for messages that are not yet indexed. This could be used to make notmuch insert simpler and more robust, as part of tag synchronization, and to improve nmbug's behavior when tags arrive before messages.