Re: [notmuch] [PATCH (rebased)] Handle message renames in mail spool

diff --git a/89/9d9fa6d82ed6863d09fb1a839327f47667ffff b/89/9d9fa6d82ed6863d09fb1a839327f47667ffff
new file mode 100644 (file)
index 0000000..b15137e
--- /dev/null
+++ b/89/9d9fa6d82ed6863d09fb1a839327f47667ffff
@@ -0,0 +1,119 @@
+Return-Path: <cworth@cworth.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id AEB17431FBD;\r
+       Thu,  3 Dec 2009 16:45:27 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id bc83K61DuFi3; Thu,  3 Dec 2009 16:45:27 -0800 (PST)\r
+Received: from yoom.home.cworth.org (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 42DB5431FAE;\r
+       Thu,  3 Dec 2009 16:45:24 -0800 (PST)\r
+Received: by yoom.home.cworth.org (Postfix, from userid 1000)\r
+       id CCBA72542FB; Thu,  3 Dec 2009 16:45:22 -0800 (PST)\r
+From: Carl Worth <cworth@cworth.org>\r
+To: Mikhail Gusarov <dottedmag@dottedmag.net>, notmuch@notmuchmail.org\r
+In-Reply-To: <1259788526-14205-1-git-send-email-dottedmag@dottedmag.net>\r
+References: <1259267025-28733-1-git-send-email-dottedmag@dottedmag.net>\r
+       <1259788526-14205-1-git-send-email-dottedmag@dottedmag.net>\r
+Date: Thu, 03 Dec 2009 16:45:22 -0800\r
+Message-ID: <87zl5zfty5.fsf@yoom.home.cworth.org>\r
+MIME-Version: 1.0\r
+Content-Type: multipart/signed; boundary="=-=-=";\r
+       micalg=pgp-sha1; protocol="application/pgp-signature"\r
+Subject: Re: [notmuch] [PATCH (rebased)] Handle message renames in mail spool\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.12\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 04 Dec 2009 00:45:27 -0000\r
+\r
+--=-=-=\r
+\r
+On Thu,  3 Dec 2009 03:15:26 +0600, Mikhail Gusarov <dottedmag@dottedmag.net> wrote:\r
+> In order to handle message renames the following changes were deemed\r
+> necessary:\r
+\r
+Hi Mikhail,\r
+\r
+Thanks for contributing this patch (twice!). I think if I had gotten to\r
+it sooner, I probably would have committed it. But now...\r
+\r
+> * Mtime check on individual files was disabled. As files may be moved around\r
+> without changing their mtime, it's necessary to parse them even if they appear\r
+> old in case old message was moved. mtime check on directories was kept as moving\r
+> files changes mtime of directory.\r
+\r
+That sounds pretty harsh. I'm having to do a lot of stat() calls already\r
+when new mail arrives. Having to also parse the message ID out of\r
+(roughly, for me) 10000 files every time sounds pretty rough. Fortunately...\r
+\r
+> Note that after applying this patch notmuch still does not handle copying files\r
+> (which is harmless, database will point to the last copy of message found during\r
+> 'notmuch new') and deleting files (which is more serious, as dangling entries\r
+> will show up in searches).\r
+\r
+Today, Keith and designed an interface that will support addition,\r
+copying, rename, and deletion of files. And it will be faster than the\r
+existing code with its mtime heuristics.\r
+\r
+The complete design is on Keith's laptop right now, and hopefully he'll\r
+appear soon with an implementation. Basically, there are only two new\r
+functions needed in the library (if we got the design right):\r
+\r
+       notmuch_directory_t\r
+       notmuch_database_read_directory (notmuch_database_t *database,\r
+                                        const char *path);\r
+\r
+       notmuch_status_t\r
+       notmuch_message_remove_filename (notmuch_message_t *message,\r
+                                         const char *filename);\r
+\r
+The notmuch_directory_t object will be used in place of the current\r
+notmuch_database_get_timestamp call in notmuch-new.c. In addition to the\r
+mtime that we currently read from the database, it will provide a list\r
+of all directories and files (along with message IDs) known to the\r
+database for a particular path. So notmuch-new can then quickly compare\r
+the results of scandir with this notmuch_directory_t object and then\r
+call notmuch_database_add_message and notmuch_message_remove_filename as\r
+appropriate.\r
+\r
+I'm leaving out details about how to ensure we don't delete a message\r
+too soon if it's actually a rename that will be seen as an added file\r
+later in the scan. Obviously the implementation will need to deal with\r
+that, (either with an additional library call for "I'm done adding\r
+files, go ahead and delete dangling messages", or by postponing all\r
+calls to remove_filename until later).\r
+\r
+Oh, and one idea is to do deletion by dropping all indexed terms, but\r
+saving the message ID and any tags in the database. That's small and is\r
+the only precious data, so might be worth holding onto "just in case".\r
+\r
+Anyway, I think we'll see code for that soon, so I'm not planning to\r
+commit the offered patch. But people really needing renames might want\r
+to use it for now, (and live with any performance implications it\r
+causes).\r
+\r
+-Carl\r
+\r
+--=-=-=\r
+Content-Type: application/pgp-signature\r
+\r
+-----BEGIN PGP SIGNATURE-----\r
+Version: GnuPG v1.4.10 (GNU/Linux)\r
+\r
+iD8DBQFLGFui6JDdNq8qSWgRArJgAJ9/r/qgcmTOXv9DZAu1y0uTAJhoigCgpmnN\r
+/uZ6RHSHFN2Ou8YPb4XSZ/I=\r
+=tSvx\r
+-----END PGP SIGNATURE-----\r
+--=-=-=--\r