Re: [PATCH v1 0/3] Improve the acquisition of text parts.

diff --git a/d8/23a4a9464224408f69fd684b71b6b23738b84f b/d8/23a4a9464224408f69fd684b71b6b23738b84f
new file mode 100644 (file)
index 0000000..6041417
--- /dev/null
+++ b/d8/23a4a9464224408f69fd684b71b6b23738b84f
@@ -0,0 +1,201 @@
+Return-Path: <markwalters1009@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id C24066DE00E8\r
+ for <notmuch@notmuchmail.org>; Sat, 26 Mar 2016 02:18:34 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.305\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.305 tagged_above=-999 required=5 tests=[AWL=0.265,\r
+  DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,\r
+ FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7,\r
+ RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001]\r
+ autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id laOczQxHPryt for <notmuch@notmuchmail.org>;\r
+ Sat, 26 Mar 2016 02:18:25 -0700 (PDT)\r
+Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com\r
+ [74.125.82.68]) by arlo.cworth.org (Postfix) with ESMTPS id 11FB26DE0005 for\r
+ <notmuch@notmuchmail.org>; Sat, 26 Mar 2016 02:18:25 -0700 (PDT)\r
+Received: by mail-wm0-f68.google.com with SMTP id p65so8739050wmp.1\r
+ for <notmuch@notmuchmail.org>; Sat, 26 Mar 2016 02:18:24 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;\r
+ h=from:to:subject:in-reply-to:references:user-agent:date:message-id\r
+ :mime-version; bh=OV6tNAA4giJd6MVRdulmNsrpFaqdi4WXbDR8yaUb1Q8=;\r
+ b=HubBKQDHgna15x9mb1zdpGiJqHHv8EdV+n/xryMJZ0R/K3BZpdi1fdtIFT6NQeEl8c\r
+ LVtxCdhdZLP9eHJ1+pkjSvqmebYl9dCVi4Tj+aD0nUyVSuipF3Nlcp14o7Ji2vInn9/R\r
+ oMbFTQMduxzaSlcOxAAhcFRGfWaI58Jeg/mgU0skmN52Y36y29h7uTHeeKL8QinHj0SK\r
+ rJy9eSY2fnFDD7okmtf2PLJmaEe4WSYKPc8l6aH+Xe9JPhJQNIfzM/E/2CtyIaLTeezc\r
+ LBbRj1d/ODVaKNZWvH42dko8TWu8vpjvTkWzaRfPTUcnMtCcxuBiiwRNgLklReX7NpB5\r
+ ewog==\r
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+ d=1e100.net; s=20130820;\r
+ h=x-gm-message-state:from:to:subject:in-reply-to:references\r
+ :user-agent:date:message-id:mime-version;\r
+ bh=OV6tNAA4giJd6MVRdulmNsrpFaqdi4WXbDR8yaUb1Q8=;\r
+ b=Ee1k6ZG9rr9WJ0lSCVGmIHPrernLuDV/VeyC4vvXzSsM+yj65giMIdtdq/NL8/teCh\r
+ 4FeX4r+CqXfSfOIRYFhh/U8rCQML55+84MQRtdyQ2rxMbey6YxhE3WIxQGJ4iyQHNtha\r
+ a6UszeRy8UltVOUvSWsNIZf0Fihr11AT7YAYraAtopQ2TYMlGzL1Bx8NfVRUKn5RsHWP\r
+ /GQRpMZoFFusxUlY+cHMdnKNwpmPzNiyPWJGYEq3nPBHEnipFaOPbr+bn9sC7rKlBLsU\r
+ UlpVoDIGqllFsth56SctJimrR2cJVvzbNi+NZmlYUZHOMySwk1COH1R6pCMzwj3WyaBk\r
+ Jcxg==\r
+X-Gm-Message-State:\r
+ AD7BkJIJbcEXnjnzGf/oYhd26eqS5kZRgOB2dcdI58oKDDjlkh26D8g5Bga8QMhGzrjxCA==\r
+X-Received: by 10.28.136.19 with SMTP id k19mr945289wmd.11.1458983903175;\r
+ Sat, 26 Mar 2016 02:18:23 -0700 (PDT)\r
+Received: from localhost (5751dfa2.skybroadband.com. [87.81.223.162])\r
+ by smtp.gmail.com with ESMTPSA id u14sm841793wmu.8.2016.03.26.02.18.21\r
+ (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\r
+ Sat, 26 Mar 2016 02:18:22 -0700 (PDT)\r
+From: Mark Walters <markwalters1009@gmail.com>\r
+To: David Bremner <david@tethera.net>, David Edmondson <dme@dme.org>,\r
+ notmuch@notmuchmail.org\r
+Subject: Re: [PATCH v1 0/3] Improve the acquisition of text parts.\r
+In-Reply-To: <87bn6h5lf3.fsf@zancas.localnet>\r
+References: <1457457179-4707-1-git-send-email-dme@dme.org>\r
+ <87ziu2s8rb.fsf@qmul.ac.uk> <m2pouxqx3e.fsf@dme.org>\r
+ <87bn6h5lf3.fsf@zancas.localnet>\r
+User-Agent: Notmuch/0.21+69~gd27d908 (http://notmuchmail.org) Emacs/24.4.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Sat, 26 Mar 2016 09:18:20 +0000\r
+Message-ID: <87zitlppgj.fsf@qmul.ac.uk>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sat, 26 Mar 2016 09:18:34 -0000\r
+\r
+\r
+Hi\r
+\r
+Sorry this email ended up rather long:\r
+\r
+Summary: I have run a test (see below) on all of the lkml part of the\r
+performance-corpus, and all the changes look expected. So this series\r
+looks good to me.\r
+\r
+First note how we do the bodypart-insertion: for a mime type of\r
+text/plain we first try the text/plain handler, then a text/* handler,\r
+and finally a */* handler until one succeeds. Before this series, when\r
+the part is application/octet-stream but is detected as text/plain, \r
+text/plain handler fails with a "bodypart insertion error" because\r
+notmuch-get-bodypart-text fails can't get the text (because it's not\r
+officially text). Thus we fall back on the */* handler and that inserts\r
+the part. \r
+\r
+With this series notmuch-get-bodypart-text succeeds and we stop.\r
+\r
+Thus in most cases the only change is that we don't get a "bodypart\r
+insertion error", but all the text looks the same. In a couple of cases\r
+the text/plain handler wraps lines/replaces ^M by unix newlines, whereas\r
+as the */* handler does not. This is an improvement.\r
+\r
+There is one more "difference" but I think this is actually something\r
+random. Sometimes when the part is application/tar or application/zip I\r
+get "Bodypart insert error: Symbol's function definition is void:\r
+gnus-recursive-directory-files". If I load gnus this goes away. In my\r
+first batch of tests this only occurred when using this series, but\r
+since then I have reproduced it on mainline. I think something else I\r
+did when setting up the test on mainline caused gnus to be loaded, but i\r
+have not worked out what is going on there.\r
+\r
+Finally, the test was as follows. I downloaded the performance corpus,\r
+configured a separate notmuch config file to use the\r
+performance-test/corpus/mail/lkml as the mailstore, went into\r
+notmuch-emacs and to the inbox (which contained all messages) and ran\r
+the following lisp function\r
+\r
+\r
+(defun my-save-all-show ()\r
+  (interactive)\r
+  (goto-char (point-min))\r
+  (let ((count 0))\r
+    (while (notmuch-search-find-thread-id)\r
+      (let ((thread-id (notmuch-search-find-thread-id)))\r
+        (setq count (1+ count))\r
+        (message "Thread %s: %s" count thread-id)\r
+        (notmuch-show thread-id)\r
+        (let ((text (buffer-string))\r
+              (coding-system-for-write 'no-conversion))\r
+          (with-temp-file (concat "OUTPUT-" thread-id) (insert text)))\r
+        (kill-buffer))\r
+      (notmuch-search-next-thread))))\r
+\r
+I moved the OUTPUT files elsewhere and repeated with this series applied\r
+and then ran diff on the output. This gave 7 threads with a change (each\r
+an individual message) from the 16000 threads/ 100000 messages which I\r
+looked at individually as above.\r
+\r
+Best wishes\r
+\r
+Mark\r
+\r
+\r
+\r
+\r
+\r
+\r
+On Mon, 14 Mar 2016, David Bremner <david@tethera.net> wrote:\r
+> David Edmondson <dme@dme.org> writes:\r
+>\r
+>> On Sun, Mar 13 2016, Mark Walters wrote:\r
+>>> However, it would be sensible to get testing in a greater variety of\r
+>>> charsets/encodings\r
+>>\r
+>> Agreed. Does anyone have suggestions on how we might achieve this? A\r
+>> corpus of mail that we could use?\r
+>\r
+> Maybe the notmuch performance corpus, particularly the lkml sample.\r
+>\r
+> grep -R charset= performance-test/corpus/mail/lkml | sed -e 's/^.*charset=//' -e 's/;.*//' -e 's/"//g' | tr '[A-Z]' '[a-z]' | sort -u\r
+>\r
+> gives\r
+>\r
+> euc-kr\r
+> gb2312\r
+> iso-2022-jp\r
+> iso-2022-jp-2\r
+> iso-8859-1\r
+> iso-8859-14\r
+> iso 8859-15\r
+> iso-8859-15\r
+> iso-8859-1\r
+> iso-8859-2\r
+> iso-8859-6\r
+> iso-8859-7\r
+> iso-8859-9\r
+> koi8-r\r
+> koi8-u\r
+> ks_c_5601-1987\r
+> shift_jis\r
+> unknown\r
+> unknown-8bit\r
+> us-ascii\r
+> utf8\r
+> utf-8\r
+> windows-1250\r
+> windows-1251\r
+> windows-1252\r
+> windows-1255\r
+>\r
+>\r
+> to unpack the corpus\r
+>\r
+> cd performance-test\r
+> make download-corpus\r
+> ./T00-new.sh --large\r
+>\r
+> probably interrupt the test once notmuch-new starts running.\r