b7/0dc879377b862c138acb14c4e7cade79a7566f

   1 Return-Path: <dkg@fifthhorseman.net>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5  by arlo.cworth.org (Postfix) with ESMTP id 7DA916DE179D\r
   6  for <notmuch@notmuchmail.org>; Wed, 10 Feb 2016 09:21:34 -0800 (PST)\r
   7 X-Virus-Scanned: Debian amavisd-new at cworth.org\r
   8 X-Spam-Flag: NO\r
   9 X-Spam-Score: -0.017\r
  10 X-Spam-Level: \r
  11 X-Spam-Status: No, score=-0.017 tagged_above=-999 required=5\r
  12  tests=[AWL=-0.017] autolearn=disabled\r
  13 Received: from arlo.cworth.org ([127.0.0.1])\r
  14  by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
  15  with ESMTP id Bl9KNKXl6Bxs for <notmuch@notmuchmail.org>;\r
  16  Wed, 10 Feb 2016 09:21:32 -0800 (PST)\r
  17 Received: from che.mayfirst.org (che.mayfirst.org [209.234.253.108])\r
  18  by arlo.cworth.org (Postfix) with ESMTP id 4569E6DE13DA\r
  19  for <notmuch@notmuchmail.org>; Wed, 10 Feb 2016 09:21:32 -0800 (PST)\r
  20 Received: from fifthhorseman.net (unknown [38.109.115.130])\r
  21  by che.mayfirst.org (Postfix) with ESMTPSA id 769BCF997;\r
  22  Wed, 10 Feb 2016 12:21:28 -0500 (EST)\r
  23 Received: by fifthhorseman.net (Postfix, from userid 1000)\r
  24  id E61661FF75; Wed, 10 Feb 2016 12:21:27 -0500 (EST)\r
  25 From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>\r
  26 To: Jameson Graef Rollins <jrollins@finestructure.net>,\r
  27  Notmuch Mail <notmuch@notmuchmail.org>\r
  28 Subject: Re: [PATCH v3 15/16] added notmuch_message_reindex\r
  29 In-Reply-To: <87oabpnzt4.fsf@alice.fifthhorseman.net>\r
  30 References: <1454272801-23623-1-git-send-email-dkg@fifthhorseman.net>\r
  31  <1454272801-23623-16-git-send-email-dkg@fifthhorseman.net>\r
  32  <87mvr9s8gy.fsf@servo.finestructure.net>\r
  33  <87oabpnzt4.fsf@alice.fifthhorseman.net>\r
  34 User-Agent: Notmuch/0.21+72~gd8c4f1c (http://notmuchmail.org) Emacs/24.5.1\r
  35  (x86_64-pc-linux-gnu)\r
  36 Date: Wed, 10 Feb 2016 12:21:24 -0500\r
  37 Message-ID: <871t8ko50r.fsf@alice.fifthhorseman.net>\r
  38 MIME-Version: 1.0\r
  39 Content-Type: multipart/signed; boundary="==-=-=";\r
  40  micalg=pgp-sha512; protocol="application/pgp-signature"\r
  41 X-BeenThere: notmuch@notmuchmail.org\r
  42 X-Mailman-Version: 2.1.20\r
  43 Precedence: list\r
  44 List-Id: "Use and development of the notmuch mail system."\r
  45  <notmuch.notmuchmail.org>\r
  46 List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
  47  <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  48 List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
  49 List-Post: <mailto:notmuch@notmuchmail.org>\r
  50 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  51 List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
  52  <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  53 X-List-Received-Date: Wed, 10 Feb 2016 17:21:34 -0000\r
  54 \r
  55 --==-=-=\r
  56 Content-Type: multipart/mixed; boundary="=-=-="\r
  57 \r
  58 --=-=-=\r
  59 Content-Type: text/plain\r
  60 Content-Transfer-Encoding: quoted-printable\r
  61 \r
  62 On Tue 2016-02-09 20:01:43 -0500, Daniel Kahn Gillmor wrote:\r
  63 >> I just wanted to mention that I think there's a problem with the reindex\r
  64 >> functionality introduced in this patch (or in 16/16).  It looks like\r
  65 >> this function irrevocably busts apart threads.  dkg and I are\r
  66 >> investigating.\r
  67 >\r
  68 > it doesn't appear to be irrevocable to me, but it is definitely doing\r
  69 > something weird with threading.\r
  70 \r
  71 OK, this is definitely tickling some problems with threading, but those\r
  72 are problems that are present already in existing versions of notmuch,\r
  73 unrelated to this series.\r
  74 \r
  75 When removing a message from the database, its earlier presence doesn't\r
  76 become a ghost message, and as a result anything that points to it\r
  77 doesn't get assembled into the prior thread properly.\r
  78 \r
  79 The attached tarball has a python test showing this behavior with a\r
  80 simple thread of two messages:\r
  81 \r
  82 0 dkg@frigg:~/src/notmuch/threading-test$ ./run-test=20\r
  83 Found 2 total files (that's not much mail).\r
  84 Processed 2 total files in almost no time.\r
  85 Added 2 new messages to the database.\r
  86 Threads: 1\r
  87 removing and re-adding a@example.com\r
  88 Threads: 2\r
  89 removing and re-adding b@example.com\r
  90 Threads: 1\r
  91 0 dkg@frigg:~/src/notmuch/threading-test$=20\r
  92 \r
  93 the relevant python function is:\r
  94 \r
  95 \r
  96 def remove_and_readd(db, mid):\r
  97     print('removing and re-adding', mid)\r
  98     m =3D db.find_message(mid)\r
  99     f =3D m.get_filename()\r
 100     db.remove_message(f)\r
 101     db.add_message(f)\r
 102 \r
 103 \r
 104 \r
 105 I think when a message is removed from the database, we need to know\r
 106 whether anything else (in its same thread?) refers to it.  If so, we\r
 107 should keep it around as a ghost message instead of fully removing it.\r
 108 \r
 109 does this sound like the right approach?\r
 110 \r
 111      --dkg\r
 112 \r
 113 \r
 114 --=-=-=\r
 115 Content-Type: application/x-gtar-compressed\r
 116 Content-Disposition: attachment; filename=threading-test.tgz\r
 117 Content-Transfer-Encoding: base64\r
 118 \r
 119 H4sIAAAAAAACA+1YW2/TMBTus3+FGUhtEUlz71YxBNoF+jAmjSEeIyc+aYOaOHOcbX3ht3OaZhO9\r
 120 DARsGRd/L0l8Lo5z/H12XLDrcDITEZuFU2AcZOf+YSGCIKiviPUrGr2O7QQ++gydwMZ22/NdtzPp\r
 121 tICqVExil1II9T2/H9nXB/eXwHdoLLIMcrVvc8+y+S5YfGjx3Thme47LbAhc5rnRrmvbzl7sscAi\r
 122 HY1/BmoqkfZpPjEUlGrwIH0s+DAc+nfzH+/X+W85dsfX/G+7/uYkVekkFxLuXf+9u+vvbeh/gDOm\r
 123 Y+n6PzhyobIqnhqxyJN0Qp5/IRmUJZtAOTAbm9b7/0f/b4v/yPrvB66l9f8R6x9XcvCI9Q88vf4/\r
 124 ev1ZS+u/42yu/37g6PW/BZwsC26M+Yiy13DNsmIGJv4Skg9V9Blihc2MkUOmYEQ/AX9BbYseQ0Qd\r
 125 LBLej+zhyN6jhuVbFjmWIkP/WRrDSqpzMaKRiFbayDuQQNOSqinQJJWlos3k0/uNP4T/UUv8d+1g\r
 126 C/99zf+W+R9t538URT/F/3WmL9i/qQlnkKAA5DGUI/pyRXlekXFunEExmxuL0HXjinCUgD8uXCvH\r
 127 ffFfgrHaYhbzB9//DYMN/g+HruZ/G3j6ZFCVKPVpPoD8khZzNRW5S0iaFUIqenME0DyKkhAOCZWQ\r
 128 iUsIWc7DxWThPR69oFnK+yNCEYVMc9Xr1l44kSj6YYiBjvjUXXrWjhndpzwykxQTNRzu3RoTNGbm\r
 129 BFSYpDPIWQa9pQEjmhe4iUluDdjHt62ER5ilGYWJKsYiVmIfgsP+eqt5cnp4ZJ4dvTkMP52Nz4/6\r
 130 5GL5ejGOUUF4UYGc97q1lHUxdTPK85ov5QjHdYEKVeUqXFKo7PX7ZOuX6q5IGub6jVTRr6S6m/9V\r
 131 Xt+0ev7roW39/993A83/dvi/4D7O/ykhcF2z/P3p+cnHg3fhwen74/Hb/We94or3B2snhURm1JAJ\r
 132 3WnMG8eGO2hazbRDiBJo2WJoYugyOS7rivIbWhZMTTe62RpSlSDNhVDQxRyuH+90Q55kTM5DyFg6\r
 133 q/1viSQkDu8mLocrQsytC6PebmhoaGhoaGhoaPxt+AoTIjhIACgAAA==\r
 134 --=-=-=--\r
 135 \r
 136 --==-=-=\r
 137 Content-Type: application/pgp-signature; name="signature.asc"\r
 138 \r
 139 -----BEGIN PGP SIGNATURE-----\r
 140 Version: GnuPG v2\r
 141 \r
 142 iQJ8BAEBCgBmBQJWu3GUXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w\r
 143 ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFREIyRTc0RjU2RkNGMkI2NzI5N0I3MzUy\r
 144 NEVDRkY1QUZGNjgzNzBBAAoJECTs/1r/aDcKJm8QAIdV/RVM0hjmNJHIGJ+zkg+t\r
 145 u4ZXt+OBZBkIVWDZ5Ksu9RrqxwG/jrfipVRp2U+GWGQh2wvsaxY6h1+rt942SEIj\r
 146 dYOxfEGsOC4Zr5YwmBZVFYzT1Ndp1gt9urKLfKzwrbKq9yW060/AOOoc02lobOIX\r
 147 rKazE9wl+scJfHDaSfpEzd+Ts5awWlXgkWd1hQTJ2z/8qndFoA+HfdA/DnwW1iI+\r
 148 advd9w8c+ZCXx/dRAGh9H3aQBD9tPShh9ceF7Szii9SzJ8SAtzO8pVF8ndMEqZ6c\r
 149 pxR/hilFhTB1FuLqf8feKflUBAGSBa0pI1ceBDqr+7mbaxS+88ZJcpyIJ3b2Hexc\r
 150 /yTPaxJPeWgQTSFyHp0WsuEU4FeZTh+tJOKL2yRLLqVKfhZ8oDcPSjzLBvbDpst9\r
 151 ytzHOTM/GpwxP+bEFr14zi4wAJANmEmJdfmFxUYJbjI4UEn7R7d6qSvnIjKpJbLa\r
 152 8tt6NobX8UWtycL6PxdYUVFwL6pAe4tmHp6b4b252Us8jR/OkMP8tYroyha0PICQ\r
 153 gfbfCvEQ9so1URDcTf6zzZ1Wkg0DG0sL10n7Ujwo7omTmLaMvHhCFxtagEOPmTgq\r
 154 mcEB/6c5ylVHDicHXTYWx/0XMvgea/NAWDud3DIXyu5dg+tUCk74vFOupusBR0ik\r
 155 L34eAydJ09C2ZQkele84\r
 156 =/P4A\r
 157 -----END PGP SIGNATURE-----\r
 158 --==-=-=--\r