e3/c81474d9b504bbe5c3e15d2d1d90af941b8eed

   1 Return-Path: <mpn@google.com>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5         by olra.theworths.org (Postfix) with ESMTP id 2F826431FB6\r
   6         for <notmuch@notmuchmail.org>; Tue,  4 Sep 2012 13:26:27 -0700 (PDT)\r
   7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
   8 X-Spam-Flag: NO\r
   9 X-Spam-Score: -0.7\r
  10 X-Spam-Level: \r
  11 X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
  12         tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7]\r
  13         autolearn=disabled\r
  14 Received: from olra.theworths.org ([127.0.0.1])\r
  15         by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
  16         with ESMTP id MfAPHEtIqKwR for <notmuch@notmuchmail.org>;\r
  17         Tue,  4 Sep 2012 13:26:26 -0700 (PDT)\r
  18 Received: from mail-ee0-f53.google.com (mail-ee0-f53.google.com\r
  19  [74.125.83.53])        (using TLSv1 with cipher RC4-SHA (128/128 bits))        (No client\r
  20  certificate requested) by olra.theworths.org (Postfix) with ESMTPS id\r
  21  10D48431FAF    for <notmuch@notmuchmail.org>; Tue,  4 Sep 2012 13:26:25 -0700\r
  22  (PDT)\r
  23 Received: by eekb47 with SMTP id b47so2986891eek.26\r
  24         for <notmuch@notmuchmail.org>; Tue, 04 Sep 2012 13:26:24 -0700 (PDT)\r
  25 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;\r
  26  s=20120113;    h=sender:from:to:subject:in-reply-to:organization:references\r
  27         :user-agent:x-face:face:x-pgp:x-pgp-fp:date:message-id:mime-version\r
  28         :content-type; bh=sNVDy4WyUKME6jjouDXoNP1/3NnfrybALBbSdrMLZO4=;\r
  29         b=YvPAp7JMv6nOnGQUBdhbqBHE/jyyursJMQ9i7pabRiF8klWSG2zzY8fwQNhBMr9PeC\r
  30         HnUorZOBkMDeMaPEqO/o1JbKLzBIuJbNOEq/1mvYf2tecXzfoutdzAxq1DEJU6gDUWVc\r
  31         +0W93MfUIqqVvGJBGKKuUnUpfj0ONasYnTj/W2UMN4X+9DOjiyYOpVzTemWCIPzayEYa\r
  32         7w9zhLDCoppXoQSUElkNABg66wxFvqfvkE8DnyLhYeZsjcH9OBtfR0qrE2veA5Vj4C+S\r
  33         VnpidoOM+PmwOAG+07n4oxzDKzu0uhjRm2Fwaf1tBje5fQUwu2BD9GjXX/mGor2puO+u    GgjQ==\r
  34 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
  35         d=google.com; s=20120113;\r
  36         h=sender:from:to:subject:in-reply-to:organization:references\r
  37         :user-agent:x-face:face:x-pgp:x-pgp-fp:date:message-id:mime-version\r
  38         :content-type:x-gm-message-state;\r
  39         bh=sNVDy4WyUKME6jjouDXoNP1/3NnfrybALBbSdrMLZO4=;\r
  40         b=hHyT9HKQ4bIjJjIaPTFNtr7YKxB1i9XJ/L5/YD2eZopmcE8OevRTmHlJ1hpM1lICZQ\r
  41         2FBAGqrHrMcRqIFsVwDshc/T7BcKQKyUugl2WKPYxEdb/24p6y1Y2A0urL2LgFnrZaDm\r
  42         DsZMnc/whEyljoD+DDsXP3OD0uGl5ClzJtRgR6MRt4hXhSiFBfSusnu2Qhr3IPA6SRQU\r
  43         m27lTrLMAl1OttXlx1pGKfXuJmKdzRTKS+3iA3tQObLblxIxe7kjvB6mOkOh7o0yxU4Z\r
  44         daQhwTn+Ai+bbOP4mog3wjRiFRMdNBuuJLqBPiUVm9qRke4r7EYZ1EOFBqL8sxztf3G8\r
  45         yfYQ==\r
  46 Received: by 10.14.218.134 with SMTP id k6mr27948267eep.14.1346790384901;\r
  47         Tue, 04 Sep 2012 13:26:24 -0700 (PDT)\r
  48 Received: by 10.14.218.134 with SMTP id k6mr27948245eep.14.1346790384652;\r
  49         Tue, 04 Sep 2012 13:26:24 -0700 (PDT)\r
  50 Received: from mpn-glaptop ([2620:0:105f:5:f2de:f1ff:fe35:1a72])\r
  51         by mx.google.com with ESMTPS id 45sm48181447eeb.8.2012.09.04.13.26.22\r
  52         (version=TLSv1/SSLv3 cipher=OTHER);\r
  53         Tue, 04 Sep 2012 13:26:23 -0700 (PDT)\r
  54 Sender: Michal Nazarewicz <mpn@google.com>\r
  55 From: Michal Nazarewicz <mina86@mina86.com>\r
  56 To: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>, notmuch@notmuchmail.org\r
  57 Subject: Re: [PATCH] Add notmuch-remove-duplicates.py script to contrib.\r
  58 In-Reply-To: <87d321sg20.fsf@gmail.com>\r
  59 Organization: http://mina86.com/\r
  60 References: <1346784785-19746-1-git-send-email-dmitry.kurochkin@gmail.com>\r
  61         <xa1tligpk1za.fsf@mina86.com> <87d321sg20.fsf@gmail.com>\r
  62 User-Agent: Notmuch/0.14+2~g416b120 (http://notmuchmail.org) Emacs/24.2.50.1\r
  63         (x86_64-unknown-linux-gnu)\r
  64 X-Face: PbkBB1w#)bOqd`iCe"Ds{e+!C7`pkC9a|f)Qo^BMQvy\q5x3?vDQJeN(DS?|-^$uMti[3D*#^_Ts"pU$jBQLq~Ud6iNwAw_r_o_4]|JO?]}P_}Nc&"p#D(ZgUb4uCNPe7~a[DbPG0T~!&c.y$Ur,=N4RT>]dNpd;        KFrfMCylc}gc??'U2j,!8%xdD\r
  65 Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWbfGlUPDDHgE57V0jUupKjgIObY0PLrom9mH4dFRK4gmjPs41MxjOgAAACQElEQVQ4jW3TMWvbQBQHcBk1xE6WyALX1069oZBMlq+ouUwpEQQ6uRjttkWP4CmBgGM0BQLBdPFZYPsyFUo6uEtKDQ7oy/U96XR2Ux8ehH/89Z6enqxBcS7Lg81jmSuujrfCZcLI/TYYvbGj+jbgFpHJ/bqQAUISj8iLyu4LuFHJTosxsucO4jSDNE0Hq3hwK/ceQ5sx97b8LcUDsILfk+ovHkOIsMbBfg43VuQ5Ln9YAGCkUdKJoXR9EclFBhixy3EGVz1K6eEkhxCAkeMMnqoAhAKwhoUJkDrCqvbecaYINlFKSRS1i12VKH1XpUd4qxL876EkMcDvHj3s5RBajHHMlA5iK32e0C7VgG0RlzFPvoYHZLRmAC0BmNcBruhkE0KsMsbEc62ZwUJDxWUdMsMhVqovoT96i/DnX/ASvz/6hbCabELLk/6FF/8PNpPCGqcZTGFcBhhAaZZDbQPaAB3+KrWWy2XgbYDNIinkdWAFcCpraDE/knwe5DBqGmgzESl1p2E4MWAz0VUPgYYzmfWb9yS4vCvgsxJriNTHoIBz5YteBvg+VGISQWUqhMiByPIPpygeDBE6elD973xWwKkEiHZAHKjhuPsFnBuArrzxtakRcISv+XMIPl4aGBUJm8Emk7qBYU8IlgNEIpiJhk/No24jHwkKTFHDWfPniR4iw5vJaw2nzSjfq2zffcE/GDjRC2dn0J0XwPAbDL84TvaFCJEU4Oml9pRyEUhR3Cl2t01AoEjRbs0sYugp14/4X5n4pU4EHHnMAAAAAElFTkSuQmCC\r
  66 X-PGP: 50751FF4\r
  67 X-PGP-FP: AC1F 5F5C D418 88F8 CC84 5858 2060 4012 5075 1FF4\r
  68 Date: Tue, 04 Sep 2012 22:26:16 +0200\r
  69 Message-ID: <xa1tipbtk00n.fsf@mina86.com>\r
  70 MIME-Version: 1.0\r
  71 Content-Type: multipart/mixed; boundary="=-=-="\r
  72 X-Gm-Message-State: ALoCoQk/xF9cupH17t4530QVOx1nvqEv5KEURZzPKFAr1FZehQJKvp2ihr10O2mg2NbAwjnv2j2jVTXYK7QNsO59WJVum/5rjfAvScIG+LE185k5oCmc3wu2Q4aJsKsBiicacCBawGDsYp9gj1eufS3q0tjCSvYThLzm2Bv2tFSVR05AqY6fD5NtrLOVZq/vW8cRUxIvYBO6zxN7mjXb+BaDxgBBIvrVKA==\r
  73 X-BeenThere: notmuch@notmuchmail.org\r
  74 X-Mailman-Version: 2.1.13\r
  75 Precedence: list\r
  76 List-Id: "Use and development of the notmuch mail system."\r
  77         <notmuch.notmuchmail.org>\r
  78 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
  79         <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  80 List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
  81 List-Post: <mailto:notmuch@notmuchmail.org>\r
  82 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  83 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
  84         <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  85 X-List-Received-Date: Tue, 04 Sep 2012 20:26:27 -0000\r
  86 \r
  87 --=-=-=\r
  88 Content-Type: text/plain; charset=utf-8\r
  89 Content-Transfer-Encoding: quoted-printable\r
  90 \r
  91 >> On Tue, Sep 04 2012, Dmitry Kurochkin wrote:\r
  92 >>> +class MailComparator:\r
  93 >>> +    """Checks if mail files are duplicates."""\r
  94 >>> +    def __init__(self, filename):\r
  95 >>> +        self.filename =3D filename\r
  96 >>> +        self.mail =3D self.readFile(self.filename)\r
  97 >>> +\r
  98 >>> +    def isDuplicate(self, filename):\r
  99 >>> +        return self.mail =3D=3D self.readFile(filename)\r
 100 >>> +\r
 101 >>> +    @staticmethod\r
 102 >>> +    def readFile(filename):\r
 103 >>> +        with open(filename) as f:\r
 104 >>> +            data =3D ""\r
 105 >>> +            while True:\r
 106 >>> +                line =3D f.readline()\r
 107 >>> +                for header in IGNORED_HEADERS:\r
 108 >>> +                    if line.startswith(header):\r
 109 \r
 110 > Michal Nazarewicz <mina86@mina86.com> writes:\r
 111 >> Case of headers should be ignored, but this does not ignore it.\r
 112 \r
 113 On Tue, Sep 04 2012, Dmitry Kurochkin wrote:\r
 114 > It does.\r
 115 \r
 116 Wait, how?  If line is =E2=80=9Creceived:=E2=80=9D how does it starts with =\r
 117 =E2=80=9CReceived:=E2=80=9D?\r
 118 \r
 119 >>> +            if os.path.realpath(comparator.filename) =3D=3D os.path.re=\r
 120 alpath(filename):\r
 121 >>> +                print "Message '%s' has filenames pointing to the\r
 122 >>> same file: '%s' '%s'" % (msg.get_message_id(), comparator.filename,\r
 123 >>> filename)\r
 124 >>\r
 125 >> So why aren't those removed?\r
 126 >>\r
 127 >\r
 128 > Because it is the same file indexed twice (probably because of\r
 129 > symlinks).  We do not want to remove the only message file.\r
 130 \r
 131 Ah, right, with symlinks this is troublesome, but than again, we can\r
 132 check if there is at least one non-symlink.  If there is, delete\r
 133 everything else, if there is not, delete all but one arbitrarily chosen\r
 134 symlink.\r
 135 \r
 136 >>> +            elif comparator.isDuplicate(filename):\r
 137 >>> +                os.remove(filename)\r
 138 >>> +                duplicates_count +=3D 1\r
 139 >>> +            else:\r
 140 >>> +                #print "Potential duplicates: %s" % msg.get_message_id=\r
 141 ()\r
 142 >>> +                suspected_duplicates_count +=3D 1\r
 143 >>> +\r
 144 >>> +    new_timestamp =3D time.time()\r
 145 >>> +    if new_timestamp - timestamp > 1:\r
 146 >>> +        timestamp =3D new_timestamp\r
 147 >>> +        sys.stdout.write("\rProcessed %s messages, removed %s duplicat=\r
 148 es..." % (msg_count, duplicates_count))\r
 149 >>> +        sys.stdout.flush()\r
 150 >>> +\r
 151 >>> +print "\rFinished. Processed %s messages, removed %s duplicates." % (m=\r
 152 sg_count, duplicates_count)\r
 153 >>> +if duplicates_count > 0:\r
 154 >>> +    print "You might want to run 'notmuch new' now."\r
 155 >>> +\r
 156 >>> +if suspected_duplicates_count > 0:\r
 157 >>> +    print\r
 158 >>> +    print "Found %s messages with duplicate IDs but different content.=\r
 159 " % suspected_duplicates_count\r
 160 >>> +    print "Perhaps we should ignore more headers."\r
 161 >>\r
 162 >> Please consider the following instead (not tested):\r
 163 \r
 164 > Thanks for reviewing my poor python code :) I am afraid I do not have\r
 165 > enough interest in improving it.  I just implemented a simple solution\r
 166 > for my problem.  Though it looks like you already took time to rewrite\r
 167 > the script.  Would be great if you send it as a proper patch obsoleting\r
 168 > this one.\r
 169 \r
 170 Bah, I'll probably won't have time to properly test it.\r
 171 \r
 172 --=20\r
 173 Best regards,                                         _     _\r
 174 .o. | Liege of Serenely Enlightened Majesty of      o' \,=3D./ `o\r
 175 ..o | Computer Science,  Micha=C5=82 =E2=80=9Cmina86=E2=80=9D Nazarewicz   =\r
 176  (o o)\r
 177 ooo +----<email/xmpp: mpn@google.com>--------------ooO--(_)--Ooo--\r
 178 --=-=-=\r
 179 Content-Type: multipart/signed; boundary="==-=-=";\r
 180         micalg=pgp-sha1; protocol="application/pgp-signature"\r
 181 \r
 182 --==-=-=\r
 183 Content-Type: text/plain\r
 184 \r
 185 \r
 186 --==-=-=\r
 187 Content-Type: application/pgp-signature\r
 188 \r
 189 -----BEGIN PGP SIGNATURE-----\r
 190 Version: GnuPG v1.4.10 (GNU/Linux)\r
 191 \r
 192 iQIbBAEBAgAGBQJQRmPoAAoJECBgQBJQdR/0kLEP+KCPbNE7PTqoYiHjOEc8QpFD\r
 193 LiKIHYNFdtx41eYbBuOMovNyBE4CS7F1WyFnDXSoXY2ajRgHFUjEwQxncakCGyD+\r
 194 OxJGUGsVWUo8Vq0Sb+cp5+a5Giz6iDU57XvUyXrqgdRZsGPpSPJVUtGpXCXSGJkX\r
 195 UA9X/Q/uUiUbZGRsLgwwRLI7NBkNMbHR8WHJBBEt2cIUPnGttRUNfhO5IVAZhr7q\r
 196 VUK06VXW6+dMWoaH4oOkkDzGOuDH41NEKXFxjtpCsKXUU0H5FG6XT5ertqGX6msB\r
 197 HMZpkSE6LYcuXMNHj4gqOtAUS7K6vao2LtLRQ0J/r8tvHCOyFeTdwcccoWZl3i8V\r
 198 sr5ZVGBWWTB3TAuRxD/ViTxH20f5EnbyoaJs1DNBQV8Df5TlqrmWl0f6WOMCs5GO\r
 199 TDN/93gF+KK1aHAVAXmsTOnkKRDYdk8NvjV8o/aoGvpvbhCVliWkARiYQFRA1X/h\r
 200 1MoHlcGDZUbJmCbhmlTun3rB8oXHfeQmqeIdmYRp5i/LwVW15TiEyw/Joa59exCi\r
 201 s3raOx7HU4Tke65S0JQ4tpTuWyBFMetmHoFH+ainb6FjGop5u6Obnl47NcxgtC5j\r
 202 yTeHT6iIgC3Y6sDnqjs7/UVH+FtDHm8nvhlBVqTacARUEsDkrScDLKuigcwQkT4E\r
 203 +5qIEIK1Qqjcl2zNCNg=\r
 204 =o9Ln\r
 205 -----END PGP SIGNATURE-----\r
 206 --==-=-=--\r
 207 \r
 208 --=-=-=--\r