From 2183f276f24570dcc60f801fc53981438bca3870 Mon Sep 17 00:00:00 2001 From: Michal Sojka Date: Sun, 26 Feb 2012 10:33:16 +0100 Subject: [PATCH] Re: Double decoded text/html parts (was: [PATCH] test: Add test for searching of uncommonly encoded messages) --- dd/e5e7f913a0772eca06eb7459c65c3d9c4d528b | 135 ++++++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 dd/e5e7f913a0772eca06eb7459c65c3d9c4d528b diff --git a/dd/e5e7f913a0772eca06eb7459c65c3d9c4d528b b/dd/e5e7f913a0772eca06eb7459c65c3d9c4d528b new file mode 100644 index 000000000..73e8cbe8f --- /dev/null +++ b/dd/e5e7f913a0772eca06eb7459c65c3d9c4d528b @@ -0,0 +1,135 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 2F259431FBF + for ; Sun, 26 Feb 2012 01:33:21 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -2.3 +X-Spam-Level: +X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id vHidQBY8fa39 for ; + Sun, 26 Feb 2012 01:33:20 -0800 (PST) +Received: from max.feld.cvut.cz (max.feld.cvut.cz [147.32.192.36]) + by olra.theworths.org (Postfix) with ESMTP id 4150F431FBD + for ; Sun, 26 Feb 2012 01:33:20 -0800 (PST) +Received: from localhost (unknown [192.168.200.4]) + by max.feld.cvut.cz (Postfix) with ESMTP id F314919F33AD; + Sun, 26 Feb 2012 10:33:18 +0100 (CET) +X-Virus-Scanned: IMAP AMAVIS +Received: from max.feld.cvut.cz ([192.168.200.1]) + by localhost (styx.feld.cvut.cz [192.168.200.4]) (amavisd-new, + port 10044) + with ESMTP id 0D5y2YkqSFhu; Sun, 26 Feb 2012 10:33:17 +0100 (CET) +Received: from imap.feld.cvut.cz (imap.feld.cvut.cz [147.32.192.34]) + by max.feld.cvut.cz (Postfix) with ESMTP id 99AFD19F33A6; + Sun, 26 Feb 2012 10:33:17 +0100 (CET) +Received: from steelpick.2x.cz (cable-86-56-3-85.cust.telecolumbus.net + [86.56.3.85]) (Authenticated sender: sojkam1) + by imap.feld.cvut.cz (Postfix) with ESMTPSA id 60887660968; + Sun, 26 Feb 2012 10:33:17 +0100 (CET) +Received: from wsh by steelpick.2x.cz with local (Exim 4.77) + (envelope-from ) + id 1S1aTk-00017s-EQ; Sun, 26 Feb 2012 10:33:16 +0100 +From: Michal Sojka +To: Serge Z , notmuch@notmuchmail.org +Subject: Re: Double decoded text/html parts (was: [PATCH] test: Add test for + searching of uncommonly encoded messages) +In-Reply-To: <20120225083600.17873.66388@localhost> +References: <877gzd5axk.fsf@steelpick.2x.cz> + <1330043595-22054-1-git-send-email-sojkam1@fel.cvut.cz> + <20120224042925.2870.87924@localhost> + <874nug67il.fsf@steelpick.2x.cz> + <20120224075700.13214.28221@localhost> + <874nugiq2g.fsf@steelpick.2x.cz> + <20120225083600.17873.66388@localhost> +User-Agent: Notmuch/0.11.1+239~g2e86bb7 (http://notmuchmail.org) Emacs/23.3.1 + (x86_64-pc-linux-gnu) +Date: Sun, 26 Feb 2012 10:33:16 +0100 +Message-ID: <871upihrc3.fsf@steelpick.2x.cz> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 26 Feb 2012 09:33:21 -0000 + +On Sat, 25 Feb 2012, Serge Z wrote: +> +> Hi! +> I've struck another problem: +> +> I've got an html/text email with body encoded with cp1251. +> Its encoding is mentioned in both Content-type: email header and html +> tag. So when the client tries to display it with external html2text converter, +> The message is decoded twice: first by client, second by html2text (I +> use w3m). + +Right. After my analysis of the problem (see below) it seems there is no +trivial solution for this. + +> As I understand, notmuch (while indexing this message) decodes it once and +> index it in the right way (though including html tags to index). But what if +> the message contains no "charset" option in Content-Type email header but +> contain content-type tag with charset noted? + +This should not happen. It violates RFC 2046, section 4.1.2. + +> Should such message be considered as being composed wrong or it should +> be indexed with diving into html details (content-type)? + +I don't think it's wrongly composed and it should be even correctly +indexed (with my patch). The problem is when you view such a message +with an external HTML viewer. + +In my mailbox I can find two different types of text/html parts. First, +the parts that contain complete HTML document including all headers and +especially . +Such parts could be passed to external HTML viewer without any decoding +by notmuch. + +The second type is text/html part that does not contain any HTML +headers. Passing such a part to an external HTML viewer undecoded would +require it to guess the correct charset from the content. + +AFAIK Firefox users can set fallback charset (used for HTML documents +with unknown charset) in the preferences, but I don't know what other +browsers would do. In particular, do you know how w3m behaves when +charset is not specified? + +In any way, if we want notmuch to do the right thing, we should analyze +the content of text/html parts and decide whether to decode the part or +not. Perhaps, a simple heuristic could be to search the content of the +part for strings "charset=" and "encoding=" and if any is found, notmuch +wouldn't decode that part. Otherwise it will decode it according to +Content-Type header. + +As a curiosity, I found the following in one of my emails. Note that two +different encodings (iso-8859-2 and windows-1250) are specified at the +same time :) That's the reason why I think that fixing the problem won't +be trivial. + +Content-Type: text/html; charset="iso-8859-2" +Content-Transfer-Encoding: 8bit + + + + + + + +Cheers, +-Michal -- 2.26.2