Return-Path: <sojka@os.inf.tu-dresden.de>
X-Original-To: notmuch@notmuchmail.org
Delivered-To: notmuch@notmuchmail.org
Received: from localhost (localhost [127.0.0.1])
	by olra.theworths.org (Postfix) with ESMTP id 2F259431FBF
	for <notmuch@notmuchmail.org>; Sun, 26 Feb 2012 01:33:21 -0800 (PST)
X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
X-Spam-Flag: NO
X-Spam-Score: -2.3
X-Spam-Level: 
X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5
	tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled
Received: from olra.theworths.org ([127.0.0.1])
	by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id vHidQBY8fa39 for <notmuch@notmuchmail.org>;
	Sun, 26 Feb 2012 01:33:20 -0800 (PST)
Received: from max.feld.cvut.cz (max.feld.cvut.cz [147.32.192.36])
	by olra.theworths.org (Postfix) with ESMTP id 4150F431FBD
	for <notmuch@notmuchmail.org>; Sun, 26 Feb 2012 01:33:20 -0800 (PST)
Received: from localhost (unknown [192.168.200.4])
	by max.feld.cvut.cz (Postfix) with ESMTP id F314919F33AD;
	Sun, 26 Feb 2012 10:33:18 +0100 (CET)
X-Virus-Scanned: IMAP AMAVIS
Received: from max.feld.cvut.cz ([192.168.200.1])
	by localhost (styx.feld.cvut.cz [192.168.200.4]) (amavisd-new,
	port 10044)
	with ESMTP id 0D5y2YkqSFhu; Sun, 26 Feb 2012 10:33:17 +0100 (CET)
Received: from imap.feld.cvut.cz (imap.feld.cvut.cz [147.32.192.34])
	by max.feld.cvut.cz (Postfix) with ESMTP id 99AFD19F33A6;
	Sun, 26 Feb 2012 10:33:17 +0100 (CET)
Received: from steelpick.2x.cz (cable-86-56-3-85.cust.telecolumbus.net
	[86.56.3.85]) (Authenticated sender: sojkam1)
	by imap.feld.cvut.cz (Postfix) with ESMTPSA id 60887660968;
	Sun, 26 Feb 2012 10:33:17 +0100 (CET)
Received: from wsh by steelpick.2x.cz with local (Exim 4.77)
	(envelope-from <sojka@os.inf.tu-dresden.de>)
	id 1S1aTk-00017s-EQ; Sun, 26 Feb 2012 10:33:16 +0100
From: Michal Sojka <sojkam1@fel.cvut.cz>
To: Serge Z <triumhiz@yandex.ru>, notmuch@notmuchmail.org
Subject: Re: Double decoded text/html parts (was: [PATCH] test: Add test for
	searching of uncommonly encoded messages)
In-Reply-To: <20120225083600.17873.66388@localhost>
References: <877gzd5axk.fsf@steelpick.2x.cz>
	<1330043595-22054-1-git-send-email-sojkam1@fel.cvut.cz>
	<20120224042925.2870.87924@localhost>
	<874nug67il.fsf@steelpick.2x.cz>
	<20120224075700.13214.28221@localhost>
	<874nugiq2g.fsf@steelpick.2x.cz>
	<20120225083600.17873.66388@localhost>
User-Agent: Notmuch/0.11.1+239~g2e86bb7 (http://notmuchmail.org) Emacs/23.3.1
	(x86_64-pc-linux-gnu)
Date: Sun, 26 Feb 2012 10:33:16 +0100
Message-ID: <871upihrc3.fsf@steelpick.2x.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-BeenThere: notmuch@notmuchmail.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: "Use and development of the notmuch mail system."
	<notmuch.notmuchmail.org>
List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
	<mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
List-Archive: <http://notmuchmail.org/pipermail/notmuch>
List-Post: <mailto:notmuch@notmuchmail.org>
List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
	<mailto:notmuch-request@notmuchmail.org?subject=subscribe>
X-List-Received-Date: Sun, 26 Feb 2012 09:33:21 -0000

On Sat, 25 Feb 2012, Serge Z wrote:
> 
> Hi!
> I've struck another problem:
> 
> I've got an html/text email with body encoded with cp1251.
> Its encoding is mentioned in both Content-type: email header and html <meta>
> tag. So when the client tries to display it with external html2text converter,
> The message is decoded twice: first by client, second by html2text (I
> use w3m).

Right. After my analysis of the problem (see below) it seems there is no
trivial solution for this.
 
> As I understand, notmuch (while indexing this message) decodes it once and
> index it in the right way (though including html tags to index). But what if
> the message contains no "charset" option in Content-Type email header but
> contain <meta> content-type tag with charset noted?

This should not happen. It violates RFC 2046, section 4.1.2.

> Should such message be considered as being composed wrong or it should
> be indexed with diving into html details (content-type)?

I don't think it's wrongly composed and it should be even correctly
indexed (with my patch). The problem is when you view such a message
with an external HTML viewer.

In my mailbox I can find two different types of text/html parts. First,
the parts that contain complete HTML document including all headers and
especially <meta http-equiv="content-type" content="text/html; ...">.
Such parts could be passed to external HTML viewer without any decoding
by notmuch.

The second type is text/html part that does not contain any HTML
headers. Passing such a part to an external HTML viewer undecoded would
require it to guess the correct charset from the content.

AFAIK Firefox users can set fallback charset (used for HTML documents
with unknown charset) in the preferences, but I don't know what other
browsers would do. In particular, do you know how w3m behaves when
charset is not specified?

In any way, if we want notmuch to do the right thing, we should analyze
the content of text/html parts and decide whether to decode the part or
not. Perhaps, a simple heuristic could be to search the content of the
part for strings "charset=" and "encoding=" and if any is found, notmuch
wouldn't decode that part. Otherwise it will decode it according to
Content-Type header.

As a curiosity, I found the following in one of my emails. Note that two
different encodings (iso-8859-2 and windows-1250) are specified at the
same time :) That's the reason why I think that fixing the problem won't
be trivial.

Content-Type: text/html; charset="iso-8859-2"
Content-Transfer-Encoding: 8bit

<?xml version="1.0" encoding="windows-1250" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-2" />

Cheers,
-Michal