Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
From: Serge Z <triumhiz@yandex.ru>
User-Agent: alot/0.21+
To: notmuch@notmuchmail.org
References: <877gzd5axk.fsf@steelpick.2x.cz>
	<1330043595-22054-1-git-send-email-sojkam1@fel.cvut.cz>
	<20120224042925.2870.87924@localhost> <874nug67il.fsf@steelpick.2x.cz>
In-Reply-To: <874nug67il.fsf@steelpick.2x.cz>
Message-ID: <20120224075700.13214.28221@localhost>
Subject: Re: [PATCH] test: Add test for searching of uncommonly encoded
	messages
Date: Fri, 24 Feb 2012 11:57:00 +0400
Precedence: list


Quoting Michal Sojka (2012-02-24 11:00:02)
>On Fri, 24 Feb 2012, Serge Z wrote:
>> =

>> Quoting Michal Sojka (2012-02-24 04:33:15)
>> >Emails that are encoded differently than as ASCII or UTF-8 are not
>> >indexed properly by notmuch. It is not possible to search for non-ASCII
>> >words within those messages.
>> =

>> Ok. But we can preprocess each incoming message right after 'getmail' to
>> convert it from html to text and to utf8 encoding. One solution is to cr=
eate a
>> seperate script for this and make gmail pipe all messages to this script=
, and
>> then to notmuch. But It would be better if maildir contains original mes=
sages
>> only, so the question is: can we make nomuch indexing engine to index
>> preprocessed message while maildir will contain original message - as it=
 was
>> obtained?
>
>Hi,
>
>I'm not big fan of adding "preprocessor". First, I thing that both
>reasons you mention are actually bugs and it would be better to fix them
>for everybody than requiring each user to configure some preprocessor.
>Second, depending on what and how would your preprocessor do, the
>initial mail indexing could be a way slower, which is also nothing that
>people want.
>
>Do you have any other use case for the preprocessor besides utf8 and
>html->text conversions?
>
>Cheers,
>-Michal

Well, I don't want to add any external preprocessor too.

This may be considered as an architectural decision: search engine should n=
ot
access messages directly, but through some preprocessing layer which would
handle the case of different encodings in body and headers, RFC2047-encoded
headers (if this is not handled yet) etc.

Anyway, this solution imho would be nice to be concluded inside a separate
library which would be useful for notmuch clients as well as other mail
indexing engines. Or an existing library should be looked for.