Re: [PATCH 1/1] Store and search for canonical Unicode text [WIP]

diff --git a/ca/43af5fe1a7b652d56f6aff17c168b1d504b3c7 b/ca/43af5fe1a7b652d56f6aff17c168b1d504b3c7
new file mode 100644 (file)
index 0000000..a7f6b64
--- /dev/null
+++ b/ca/43af5fe1a7b652d56f6aff17c168b1d504b3c7
@@ -0,0 +1,102 @@
+Return-Path: <david@tethera.net>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id 1E9AD6DE1B56\r
+ for <notmuch@notmuchmail.org>; Wed,  2 Sep 2015 18:13:25 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0.114\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0.114 tagged_above=-999 required=5 tests=[AWL=0.114]\r
+ autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id hgFK-dBBJ5Bc for <notmuch@notmuchmail.org>;\r
+ Wed,  2 Sep 2015 18:13:23 -0700 (PDT)\r
+Received: from gitolite.debian.net (gitolite.debian.net [87.98.215.224])\r
+ by arlo.cworth.org (Postfix) with ESMTPS id D37D96DE1B51\r
+ for <notmuch@notmuchmail.org>; Wed,  2 Sep 2015 18:13:22 -0700 (PDT)\r
+Received: from remotemail by gitolite.debian.net with local (Exim 4.80)\r
+ (envelope-from <david@tethera.net>)\r
+ id 1ZXJ4s-0004iZ-U2; Thu, 03 Sep 2015 01:12:34 +0000\r
+Received: (nullmailer pid 17442 invoked by uid 1000); Thu, 03 Sep 2015\r
+ 01:12:14 -0000\r
+From: David Bremner <david@tethera.net>\r
+To: Rob Browning <rlb@defaultvalue.org>, notmuch@notmuchmail.org\r
+Subject: Re: [PATCH 1/1] Store and search for canonical Unicode text [WIP]\r
+In-Reply-To: <1440951676-17286-1-git-send-email-rlb@defaultvalue.org>\r
+References: <1440951676-17286-1-git-send-email-rlb@defaultvalue.org>\r
+User-Agent: Notmuch/0.20.2+60~gcb08a2e (http://notmuchmail.org) Emacs/24.5.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Wed, 02 Sep 2015 22:12:14 -0300\r
+Message-ID: <87fv2we26p.fsf@maritornes.cs.unb.ca>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Transfer-Encoding: quoted-printable\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.18\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Thu, 03 Sep 2015 01:13:25 -0000\r
+\r
+Rob Browning <rlb@defaultvalue.org> writes:\r
+\r
+>\r
+> Before this change, notmuch would index two strings that differ only\r
+> with respect to canonicalization, like to=CC=81ken and t=C3=B3ken, as sep=\r
+arate\r
+> terms, even though they may be visually indistinguishable, and do (for\r
+> most purposes) represent the same text.  After indexing, searching for\r
+> one would not find the other, and which one you present to notmuch\r
+> when you search depends on your tools.  See test/T570-normalization.sh\r
+> for a working example.\r
+\r
+One way to break this up into more bite sized pieces would be to first\r
+create one or more tests that fail with current notmuch, and mark those\r
+as broken.\r
+\r
+> Up to now, notmuch has let Xapian handle converting the incoming bytes\r
+> to UTF-8.  Xapian treats any byte sequence as UTF-8, and interprets\r
+> any invalid UTF-8 bytes as Latin-1.  This patch maintains the existing\r
+> behavior (excepting the new canonicalization) by using Xapian's\r
+> Utf8Iterator to handle the initial Unicode character parsing.\r
+\r
+Can you explain why notmuch is the right place to do this, and not\r
+Xapian? I know we talked back and forth about this, but I never really\r
+got a solid sense of what the conclusion was. Is it just dependencies?\r
+\r
+> And because when the input is already UTF-8, it just blindly converts\r
+> from UTF-8 to Unicode code points, and then back to UTF-8 (after\r
+> canonicalization), during each pass.  There are certainly\r
+> opportunities to optimize, though it may be worth discussing the\r
+> detection of data encodings more broadly first.\r
+\r
+It seems plausible to specify UTF-8 input for the library, but what\r
+about the CLI? It seems like the canonicalization operation increases\r
+the chance of mangling user input in non-UTF-8 locales.\r
+\r
+> FIXME: what about existing indexed text?\r
+\r
+I suppose some upgrade code to canonicalize all the terms? That sounds\r
+pretty slow.\r
+\r
+> ---\r
+>\r
+>  Posted for preliminary discussion, and as a milestone (it appears to\r
+>  mostly work now).  Though I doubt I'm handling things correctly\r
+>  everywhere notmuch-wise, wrt talloc, etc.\r
+\r
+I really didn't look at the code very closely, but there were a\r
+surprising number of calls to talloc_free. But those kind of details can\r
+wait.\r
+\r
+\r