From: Michal Nazarewicz Date: Tue, 4 Sep 2012 20:26:16 +0000 (+0200) Subject: Re: [PATCH] Add notmuch-remove-duplicates.py script to contrib. X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=0ef8cd4463e1938e9a10ac4c8c2fee59069a5f28;p=notmuch-archives.git Re: [PATCH] Add notmuch-remove-duplicates.py script to contrib. --- diff --git a/e3/c81474d9b504bbe5c3e15d2d1d90af941b8eed b/e3/c81474d9b504bbe5c3e15d2d1d90af941b8eed new file mode 100644 index 000000000..95bb05dfb --- /dev/null +++ b/e3/c81474d9b504bbe5c3e15d2d1d90af941b8eed @@ -0,0 +1,208 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 2F826431FB6 + for ; Tue, 4 Sep 2012 13:26:27 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id MfAPHEtIqKwR for ; + Tue, 4 Sep 2012 13:26:26 -0700 (PDT) +Received: from mail-ee0-f53.google.com (mail-ee0-f53.google.com + [74.125.83.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client + certificate requested) by olra.theworths.org (Postfix) with ESMTPS id + 10D48431FAF for ; Tue, 4 Sep 2012 13:26:25 -0700 + (PDT) +Received: by eekb47 with SMTP id b47so2986891eek.26 + for ; Tue, 04 Sep 2012 13:26:24 -0700 (PDT) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; + s=20120113; h=sender:from:to:subject:in-reply-to:organization:references + :user-agent:x-face:face:x-pgp:x-pgp-fp:date:message-id:mime-version + :content-type; bh=sNVDy4WyUKME6jjouDXoNP1/3NnfrybALBbSdrMLZO4=; + b=YvPAp7JMv6nOnGQUBdhbqBHE/jyyursJMQ9i7pabRiF8klWSG2zzY8fwQNhBMr9PeC + HnUorZOBkMDeMaPEqO/o1JbKLzBIuJbNOEq/1mvYf2tecXzfoutdzAxq1DEJU6gDUWVc + +0W93MfUIqqVvGJBGKKuUnUpfj0ONasYnTj/W2UMN4X+9DOjiyYOpVzTemWCIPzayEYa + 7w9zhLDCoppXoQSUElkNABg66wxFvqfvkE8DnyLhYeZsjcH9OBtfR0qrE2veA5Vj4C+S + VnpidoOM+PmwOAG+07n4oxzDKzu0uhjRm2Fwaf1tBje5fQUwu2BD9GjXX/mGor2puO+u GgjQ== +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=google.com; s=20120113; + h=sender:from:to:subject:in-reply-to:organization:references + :user-agent:x-face:face:x-pgp:x-pgp-fp:date:message-id:mime-version + :content-type:x-gm-message-state; + bh=sNVDy4WyUKME6jjouDXoNP1/3NnfrybALBbSdrMLZO4=; + b=hHyT9HKQ4bIjJjIaPTFNtr7YKxB1i9XJ/L5/YD2eZopmcE8OevRTmHlJ1hpM1lICZQ + 2FBAGqrHrMcRqIFsVwDshc/T7BcKQKyUugl2WKPYxEdb/24p6y1Y2A0urL2LgFnrZaDm + DsZMnc/whEyljoD+DDsXP3OD0uGl5ClzJtRgR6MRt4hXhSiFBfSusnu2Qhr3IPA6SRQU + m27lTrLMAl1OttXlx1pGKfXuJmKdzRTKS+3iA3tQObLblxIxe7kjvB6mOkOh7o0yxU4Z + daQhwTn+Ai+bbOP4mog3wjRiFRMdNBuuJLqBPiUVm9qRke4r7EYZ1EOFBqL8sxztf3G8 + yfYQ== +Received: by 10.14.218.134 with SMTP id k6mr27948267eep.14.1346790384901; + Tue, 04 Sep 2012 13:26:24 -0700 (PDT) +Received: by 10.14.218.134 with SMTP id k6mr27948245eep.14.1346790384652; + Tue, 04 Sep 2012 13:26:24 -0700 (PDT) +Received: from mpn-glaptop ([2620:0:105f:5:f2de:f1ff:fe35:1a72]) + by mx.google.com with ESMTPS id 45sm48181447eeb.8.2012.09.04.13.26.22 + (version=TLSv1/SSLv3 cipher=OTHER); + Tue, 04 Sep 2012 13:26:23 -0700 (PDT) +Sender: Michal Nazarewicz +From: Michal Nazarewicz +To: Dmitry Kurochkin , notmuch@notmuchmail.org +Subject: Re: [PATCH] Add notmuch-remove-duplicates.py script to contrib. +In-Reply-To: <87d321sg20.fsf@gmail.com> +Organization: http://mina86.com/ +References: <1346784785-19746-1-git-send-email-dmitry.kurochkin@gmail.com> + <87d321sg20.fsf@gmail.com> +User-Agent: Notmuch/0.14+2~g416b120 (http://notmuchmail.org) Emacs/24.2.50.1 + (x86_64-unknown-linux-gnu) +X-Face: PbkBB1w#)bOqd`iCe"Ds{e+!C7`pkC9a|f)Qo^BMQvy\q5x3?vDQJeN(DS?|-^$uMti[3D*#^_Ts"pU$jBQLq~Ud6iNwAw_r_o_4]|JO?]}P_}Nc&"p#D(ZgUb4uCNPe7~a[DbPG0T~!&c.y$Ur,=N4RT>]dNpd; KFrfMCylc}gc??'U2j,!8%xdD +Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWbfGlUPDDHgE57V0jUupKjgIObY0PLrom9mH4dFRK4gmjPs41MxjOgAAACQElEQVQ4jW3TMWvbQBQHcBk1xE6WyALX1069oZBMlq+ouUwpEQQ6uRjttkWP4CmBgGM0BQLBdPFZYPsyFUo6uEtKDQ7oy/U96XR2Ux8ehH/89Z6enqxBcS7Lg81jmSuujrfCZcLI/TYYvbGj+jbgFpHJ/bqQAUISj8iLyu4LuFHJTosxsucO4jSDNE0Hq3hwK/ceQ5sx97b8LcUDsILfk+ovHkOIsMbBfg43VuQ5Ln9YAGCkUdKJoXR9EclFBhixy3EGVz1K6eEkhxCAkeMMnqoAhAKwhoUJkDrCqvbecaYINlFKSRS1i12VKH1XpUd4qxL876EkMcDvHj3s5RBajHHMlA5iK32e0C7VgG0RlzFPvoYHZLRmAC0BmNcBruhkE0KsMsbEc62ZwUJDxWUdMsMhVqovoT96i/DnX/ASvz/6hbCabELLk/6FF/8PNpPCGqcZTGFcBhhAaZZDbQPaAB3+KrWWy2XgbYDNIinkdWAFcCpraDE/knwe5DBqGmgzESl1p2E4MWAz0VUPgYYzmfWb9yS4vCvgsxJriNTHoIBz5YteBvg+VGISQWUqhMiByPIPpygeDBE6elD973xWwKkEiHZAHKjhuPsFnBuArrzxtakRcISv+XMIPl4aGBUJm8Emk7qBYU8IlgNEIpiJhk/No24jHwkKTFHDWfPniR4iw5vJaw2nzSjfq2zffcE/GDjRC2dn0J0XwPAbDL84TvaFCJEU4Oml9pRyEUhR3Cl2t01AoEjRbs0sYugp14/4X5n4pU4EHHnMAAAAAElFTkSuQmCC +X-PGP: 50751FF4 +X-PGP-FP: AC1F 5F5C D418 88F8 CC84 5858 2060 4012 5075 1FF4 +Date: Tue, 04 Sep 2012 22:26:16 +0200 +Message-ID: +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="=-=-=" +X-Gm-Message-State: ALoCoQk/xF9cupH17t4530QVOx1nvqEv5KEURZzPKFAr1FZehQJKvp2ihr10O2mg2NbAwjnv2j2jVTXYK7QNsO59WJVum/5rjfAvScIG+LE185k5oCmc3wu2Q4aJsKsBiicacCBawGDsYp9gj1eufS3q0tjCSvYThLzm2Bv2tFSVR05AqY6fD5NtrLOVZq/vW8cRUxIvYBO6zxN7mjXb+BaDxgBBIvrVKA== +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 04 Sep 2012 20:26:27 -0000 + +--=-=-= +Content-Type: text/plain; charset=utf-8 +Content-Transfer-Encoding: quoted-printable + +>> On Tue, Sep 04 2012, Dmitry Kurochkin wrote: +>>> +class MailComparator: +>>> + """Checks if mail files are duplicates.""" +>>> + def __init__(self, filename): +>>> + self.filename =3D filename +>>> + self.mail =3D self.readFile(self.filename) +>>> + +>>> + def isDuplicate(self, filename): +>>> + return self.mail =3D=3D self.readFile(filename) +>>> + +>>> + @staticmethod +>>> + def readFile(filename): +>>> + with open(filename) as f: +>>> + data =3D "" +>>> + while True: +>>> + line =3D f.readline() +>>> + for header in IGNORED_HEADERS: +>>> + if line.startswith(header): + +> Michal Nazarewicz writes: +>> Case of headers should be ignored, but this does not ignore it. + +On Tue, Sep 04 2012, Dmitry Kurochkin wrote: +> It does. + +Wait, how? If line is =E2=80=9Creceived:=E2=80=9D how does it starts with = +=E2=80=9CReceived:=E2=80=9D? + +>>> + if os.path.realpath(comparator.filename) =3D=3D os.path.re= +alpath(filename): +>>> + print "Message '%s' has filenames pointing to the +>>> same file: '%s' '%s'" % (msg.get_message_id(), comparator.filename, +>>> filename) +>> +>> So why aren't those removed? +>> +> +> Because it is the same file indexed twice (probably because of +> symlinks). We do not want to remove the only message file. + +Ah, right, with symlinks this is troublesome, but than again, we can +check if there is at least one non-symlink. If there is, delete +everything else, if there is not, delete all but one arbitrarily chosen +symlink. + +>>> + elif comparator.isDuplicate(filename): +>>> + os.remove(filename) +>>> + duplicates_count +=3D 1 +>>> + else: +>>> + #print "Potential duplicates: %s" % msg.get_message_id= +() +>>> + suspected_duplicates_count +=3D 1 +>>> + +>>> + new_timestamp =3D time.time() +>>> + if new_timestamp - timestamp > 1: +>>> + timestamp =3D new_timestamp +>>> + sys.stdout.write("\rProcessed %s messages, removed %s duplicat= +es..." % (msg_count, duplicates_count)) +>>> + sys.stdout.flush() +>>> + +>>> +print "\rFinished. Processed %s messages, removed %s duplicates." % (m= +sg_count, duplicates_count) +>>> +if duplicates_count > 0: +>>> + print "You might want to run 'notmuch new' now." +>>> + +>>> +if suspected_duplicates_count > 0: +>>> + print +>>> + print "Found %s messages with duplicate IDs but different content.= +" % suspected_duplicates_count +>>> + print "Perhaps we should ignore more headers." +>> +>> Please consider the following instead (not tested): + +> Thanks for reviewing my poor python code :) I am afraid I do not have +> enough interest in improving it. I just implemented a simple solution +> for my problem. Though it looks like you already took time to rewrite +> the script. Would be great if you send it as a proper patch obsoleting +> this one. + +Bah, I'll probably won't have time to properly test it. + +--=20 +Best regards, _ _ +.o. | Liege of Serenely Enlightened Majesty of o' \,=3D./ `o +..o | Computer Science, Micha=C5=82 =E2=80=9Cmina86=E2=80=9D Nazarewicz = + (o o) +ooo +------------------ooO--(_)--Ooo-- +--=-=-= +Content-Type: multipart/signed; boundary="==-=-="; + micalg=pgp-sha1; protocol="application/pgp-signature" + +--==-=-= +Content-Type: text/plain + + +--==-=-= +Content-Type: application/pgp-signature + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.10 (GNU/Linux) + +iQIbBAEBAgAGBQJQRmPoAAoJECBgQBJQdR/0kLEP+KCPbNE7PTqoYiHjOEc8QpFD +LiKIHYNFdtx41eYbBuOMovNyBE4CS7F1WyFnDXSoXY2ajRgHFUjEwQxncakCGyD+ +OxJGUGsVWUo8Vq0Sb+cp5+a5Giz6iDU57XvUyXrqgdRZsGPpSPJVUtGpXCXSGJkX +UA9X/Q/uUiUbZGRsLgwwRLI7NBkNMbHR8WHJBBEt2cIUPnGttRUNfhO5IVAZhr7q +VUK06VXW6+dMWoaH4oOkkDzGOuDH41NEKXFxjtpCsKXUU0H5FG6XT5ertqGX6msB +HMZpkSE6LYcuXMNHj4gqOtAUS7K6vao2LtLRQ0J/r8tvHCOyFeTdwcccoWZl3i8V +sr5ZVGBWWTB3TAuRxD/ViTxH20f5EnbyoaJs1DNBQV8Df5TlqrmWl0f6WOMCs5GO +TDN/93gF+KK1aHAVAXmsTOnkKRDYdk8NvjV8o/aoGvpvbhCVliWkARiYQFRA1X/h +1MoHlcGDZUbJmCbhmlTun3rB8oXHfeQmqeIdmYRp5i/LwVW15TiEyw/Joa59exCi +s3raOx7HU4Tke65S0JQ4tpTuWyBFMetmHoFH+ainb6FjGop5u6Obnl47NcxgtC5j +yTeHT6iIgC3Y6sDnqjs7/UVH+FtDHm8nvhlBVqTacARUEsDkrScDLKuigcwQkT4E ++5qIEIK1Qqjcl2zNCNg= +=o9Ln +-----END PGP SIGNATURE----- +--==-=-=-- + +--=-=-=--