From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
To: Michal Nazarewicz <mina86@mina86.com>, notmuch@notmuchmail.org
Subject: Re: [PATCH] Add notmuch-remove-duplicates.py script to contrib.
In-Reply-To: <xa1tipbtk00n.fsf@mina86.com>
References: <1346784785-19746-1-git-send-email-dmitry.kurochkin@gmail.com>
	<xa1tligpk1za.fsf@mina86.com> <87d321sg20.fsf@gmail.com>
	<xa1tipbtk00n.fsf@mina86.com>
User-Agent: Notmuch/0.14+18~g79a73cd (http://notmuchmail.org) Emacs/23.4.1
	(x86_64-pc-linux-gnu)
Date: Wed, 05 Sep 2012 00:33:10 +0400
Message-ID: <87a9x5sf3t.fsf@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list

Michal Nazarewicz <mina86@mina86.com> writes:

>>> On Tue, Sep 04 2012, Dmitry Kurochkin wrote:
>>>> +class MailComparator:
>>>> +    """Checks if mail files are duplicates."""
>>>> +    def __init__(self, filename):
>>>> +        self.filename =3D filename
>>>> +        self.mail =3D self.readFile(self.filename)
>>>> +
>>>> +    def isDuplicate(self, filename):
>>>> +        return self.mail =3D=3D self.readFile(filename)
>>>> +
>>>> +    @staticmethod
>>>> +    def readFile(filename):
>>>> +        with open(filename) as f:
>>>> +            data =3D ""
>>>> +            while True:
>>>> +                line =3D f.readline()
>>>> +                for header in IGNORED_HEADERS:
>>>> +                    if line.startswith(header):
>
>> Michal Nazarewicz <mina86@mina86.com> writes:
>>> Case of headers should be ignored, but this does not ignore it.
>
> On Tue, Sep 04 2012, Dmitry Kurochkin wrote:
>> It does.
>
> Wait, how?  If line is =E2=80=9Creceived:=E2=80=9D how does it starts wit=
h =E2=80=9CReceived:=E2=80=9D?
>

Sorry, I misunderstood your comment.  It does not ignore the case indeed.

>>>> +            if os.path.realpath(comparator.filename) =3D=3D os.path.r=
ealpath(filename):
>>>> +                print "Message '%s' has filenames pointing to the
>>>> same file: '%s' '%s'" % (msg.get_message_id(), comparator.filename,
>>>> filename)
>>>
>>> So why aren't those removed?
>>>
>>
>> Because it is the same file indexed twice (probably because of
>> symlinks).  We do not want to remove the only message file.
>
> Ah, right, with symlinks this is troublesome, but than again, we can
> check if there is at least one non-symlink.  If there is, delete
> everything else, if there is not, delete all but one arbitrarily chosen
> symlink.
>

Sure, we could do that.

>>>> +            elif comparator.isDuplicate(filename):
>>>> +                os.remove(filename)
>>>> +                duplicates_count +=3D 1
>>>> +            else:
>>>> +                #print "Potential duplicates: %s" % msg.get_message_i=
d()
>>>> +                suspected_duplicates_count +=3D 1
>>>> +
>>>> +    new_timestamp =3D time.time()
>>>> +    if new_timestamp - timestamp > 1:
>>>> +        timestamp =3D new_timestamp
>>>> +        sys.stdout.write("\rProcessed %s messages, removed %s duplica=
tes..." % (msg_count, duplicates_count))
>>>> +        sys.stdout.flush()
>>>> +
>>>> +print "\rFinished. Processed %s messages, removed %s duplicates." % (=
msg_count, duplicates_count)
>>>> +if duplicates_count > 0:
>>>> +    print "You might want to run 'notmuch new' now."
>>>> +
>>>> +if suspected_duplicates_count > 0:
>>>> +    print
>>>> +    print "Found %s messages with duplicate IDs but different content=
." % suspected_duplicates_count
>>>> +    print "Perhaps we should ignore more headers."
>>>
>>> Please consider the following instead (not tested):
>
>> Thanks for reviewing my poor python code :) I am afraid I do not have
>> enough interest in improving it.  I just implemented a simple solution
>> for my problem.  Though it looks like you already took time to rewrite
>> the script.  Would be great if you send it as a proper patch obsoleting
>> this one.
>
> Bah, I'll probably won't have time to properly test it.
>

Same problem :)

Regards,
  Dmitry

> --=20
> Best regards,                                         _     _
> .o. | Liege of Serenely Enlightened Majesty of      o' \,=3D./ `o
> ..o | Computer Science,  Micha=C5=82 =E2=80=9Cmina86=E2=80=9D Nazarewicz =
   (o o)
> ooo +----<email/xmpp: mpn@google.com>--------------ooO--(_)--Ooo--