--- /dev/null
+Return-Path: <kas@node.shutemov.name>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 1AE76431FC3\r
+ for <notmuch@notmuchmail.org>; Tue, 17 Dec 2013 10:10:07 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id IuP8oTvzApcf for <notmuch@notmuchmail.org>;\r
+ Tue, 17 Dec 2013 10:09:59 -0800 (PST)\r
+X-Greylist: delayed 383 seconds by postgrey-1.32 at olra;\r
+ Tue, 17 Dec 2013 10:09:58 PST\r
+Received: from jenni2.inet.fi (mta-out.inet.fi [195.156.147.13])\r
+ by olra.theworths.org (Postfix) with ESMTP id E6CB3431FBF\r
+ for <notmuch@notmuchmail.org>; Tue, 17 Dec 2013 10:09:58 -0800 (PST)\r
+Received: from node.shutemov.name (80.220.224.16) by jenni2.inet.fi\r
+ (8.5.140.03) id 52775C9903BE6DDE; Tue, 17 Dec 2013 20:03:26 +0200\r
+Received: by node.shutemov.name (Postfix, from userid 1000)\r
+ id 29749417EE; Tue, 17 Dec 2013 20:03:23 +0200 (EET)\r
+Date: Tue, 17 Dec 2013 20:03:22 +0200\r
+From: "Kirill A. Shutemov" <kirill@shutemov.name>\r
+To: Jani Nikula <jani@nikula.org>\r
+Subject: Re: [PATCH] lib: Add a new prefix "list" to the search-terms syntax\r
+Message-ID: <20131217180322.GA9272@node.dhcp.inet.fi>\r
+References: <20130409083010.GA27675@raorn.name>\r
+ <1365549369-12776-1-git-send-email-raorn@raorn.name>\r
+ <87bo2ougmb.fsf@nikula.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=iso-8859-1\r
+Content-Disposition: inline\r
+Content-Transfer-Encoding: 8bit\r
+In-Reply-To: <87bo2ougmb.fsf@nikula.org>\r
+User-Agent: Mutt/1.5.22.1-rc1 (2013-10-16)\r
+Cc: notmuch@notmuchmail.org, "Alexey I. Froloff" <raorn@raorn.name>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 17 Dec 2013 18:10:07 -0000\r
+\r
+On Thu, Oct 17, 2013 at 05:17:00PM +0300, Jani Nikula wrote:\r
+> On Wed, 10 Apr 2013, "Alexey I. Froloff" <raorn@raorn.name> wrote:\r
+> > From: "Alexey I. Froloff" <raorn@raorn.name>\r
+> >\r
+> > Add support for indexing and searching the message's List-Id header.\r
+> > This is useful when matching all the messages belonging to a particular\r
+> > mailing list.\r
+> \r
+> There's an issue with our duplicate message-id handling that is likely\r
+> to cause confusion with List-Id: searches. If you receive several\r
+> duplicates of the same message (judged by the message-id), only the\r
+> first one of them gets indexed, and the rest are ignored. This means\r
+> that for messages you receive both directly and through a list, it will\r
+> be arbitrary whether the List-Id: gets indexed or not. Therefore a list:\r
+> search might not return all the messages you'd expect.\r
+\r
+I've tried to address this. The patch also adds few tests for the feature.\r
+\r
+There's still missing functionality: re-indexing existing messages for\r
+list-id, handling message removal, etc.\r
+\r
+Any comments?\r
+\r
+diff --git a/lib/database.cc b/lib/database.cc\r
+index f395061e3a73..196243e15d1a 100644\r
+--- a/lib/database.cc\r
++++ b/lib/database.cc\r
+@@ -205,6 +205,7 @@ static prefix_t BOOLEAN_PREFIX_INTERNAL[] = {\r
+ };\r
+ \r
+ static prefix_t BOOLEAN_PREFIX_EXTERNAL[] = {\r
++ { "list", "XLIST"},\r
+ { "thread", "G" },\r
+ { "tag", "K" },\r
+ { "is", "K" },\r
+@@ -2025,10 +2026,13 @@ notmuch_database_add_message (notmuch_database_t *notmuch,\r
+ date = notmuch_message_file_get_header (message_file, "date");\r
+ _notmuch_message_set_header_values (message, date, from, subject);\r
+ \r
+- ret = _notmuch_message_index_file (message, filename);\r
++ ret = _notmuch_message_index_file (message, filename, false);\r
+ if (ret)\r
+ goto DONE;\r
+ } else {\r
++ ret = _notmuch_message_index_file (message, filename, true);\r
++ if (ret)\r
++ goto DONE;\r
+ ret = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;\r
+ }\r
+ \r
+diff --git a/lib/index.cc b/lib/index.cc\r
+index 78c18cf36d10..9fe1ad6502ed 100644\r
+--- a/lib/index.cc\r
++++ b/lib/index.cc\r
+@@ -304,6 +304,47 @@ _index_address_list (notmuch_message_t *message,\r
+ }\r
+ }\r
+ \r
++static void\r
++_index_list_id (notmuch_message_t *message,\r
++ const char *list_id_header)\r
++{\r
++ const char *begin_list_id, *end_list_id, *list_id;\r
++ void *local;\r
++\r
++ if (list_id_header == NULL)\r
++ return;\r
++\r
++ /* RFC2919 says that the list-id is found at the end of the header\r
++ * and enclosed between angle brackets. If we cannot find a\r
++ * matching pair of brackets containing at least one character,\r
++ * we ignore the list id header. */\r
++ begin_list_id = strrchr (list_id_header, '<');\r
++ if (!begin_list_id) {\r
++ fprintf (stderr, "Warning: Not indexing mailformed List-Id tag.\n");\r
++ return;\r
++ }\r
++\r
++ end_list_id = strrchr(begin_list_id, '>');\r
++ if (!end_list_id || (end_list_id - begin_list_id < 2)) {\r
++ fprintf (stderr, "Warning: Not indexing mailformed List-Id tag.\n");\r
++ return;\r
++ }\r
++\r
++ local = talloc_new (message);\r
++\r
++ /* We extract the list id between the angle brackets */\r
++ list_id = talloc_strndup (local, begin_list_id + 1,\r
++ end_list_id - begin_list_id - 1);\r
++\r
++ /* _notmuch_message_add_term() may return\r
++ * NOTMUCH_PRIVATE_STATUS_TERM_TOO_LONG here. We can't fix it, but\r
++ * this is not a reason to exit with error... */\r
++ if (_notmuch_message_add_term (message, "list", list_id))\r
++ fprintf (stderr, "Warning: Not indexing List-Id: <%s>\n", list_id);\r
++\r
++ talloc_free (local);\r
++}\r
++\r
+ /* Callback to generate terms for each mime part of a message. */\r
+ static void\r
+ _index_mime_part (notmuch_message_t *message,\r
+@@ -425,14 +466,15 @@ _index_mime_part (notmuch_message_t *message,\r
+ \r
+ notmuch_status_t\r
+ _notmuch_message_index_file (notmuch_message_t *message,\r
+- const char *filename)\r
++ const char *filename,\r
++ notmuch_bool_t duplicate)\r
+ {\r
+ GMimeStream *stream = NULL;\r
+ GMimeParser *parser = NULL;\r
+ GMimeMessage *mime_message = NULL;\r
+ InternetAddressList *addresses;\r
+ FILE *file = NULL;\r
+- const char *from, *subject;\r
++ const char *from, *subject, *list_id;\r
+ notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;\r
+ static int initialized = 0;\r
+ char from_buf[5];\r
+@@ -485,6 +527,9 @@ mboxes is deprecated and may be removed in the future.\n", filename);\r
+ \r
+ from = g_mime_message_get_sender (mime_message);\r
+ \r
++ if (duplicate)\r
++ goto DUP;\r
++\r
+ addresses = internet_address_list_parse_string (from);\r
+ if (addresses) {\r
+ _index_address_list (message, "from", addresses);\r
+@@ -502,6 +547,10 @@ mboxes is deprecated and may be removed in the future.\n", filename);\r
+ \r
+ _index_mime_part (message, g_mime_message_get_mime_part (mime_message));\r
+ \r
++ DUP:\r
++ list_id = g_mime_object_get_header (GMIME_OBJECT (mime_message), "List-Id");\r
++ _index_list_id (message, list_id);\r
++\r
+ DONE:\r
+ if (mime_message)\r
+ g_object_unref (mime_message);\r
+diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h\r
+index af185c7c5ba8..138dfa58efc8 100644\r
+--- a/lib/notmuch-private.h\r
++++ b/lib/notmuch-private.h\r
+@@ -322,7 +322,8 @@ notmuch_message_get_author (notmuch_message_t *message);\r
+ \r
+ notmuch_status_t\r
+ _notmuch_message_index_file (notmuch_message_t *message,\r
+- const char *filename);\r
++ const char *filename,\r
++ notmuch_bool_t duplicate);\r
+ \r
+ /* message-file.c */\r
+ \r
+diff --git a/man/man7/notmuch-search-terms.7 b/man/man7/notmuch-search-terms.7\r
+index f1627b3488f8..29b30b7b0b00 100644\r
+--- a/man/man7/notmuch-search-terms.7\r
++++ b/man/man7/notmuch-search-terms.7\r
+@@ -52,6 +52,8 @@ terms to match against specific portions of an email, (where\r
+ \r
+ thread:<thread-id>\r
+ \r
++ list:<list-id>\r
++\r
+ folder:<directory-path>\r
+ \r
+ date:<since>..<until>\r
+@@ -109,6 +111,12 @@ within a matching directory. Only the directory components below the\r
+ top-level mail database path are available to be searched.\r
+ \r
+ The\r
++.BR list: ,\r
++is used to match mailing list ID of an email message \- contents of the\r
++List\-Id: header without the '<', '>' delimiters or decoded list\r
++description.\r
++\r
++The\r
+ .B date:\r
+ prefix can be used to restrict the results to only messages within a\r
+ particular time range (based on the Date: header) with a range syntax\r
+diff --git a/test/corpus/cur/18:2, b/test/corpus/cur/18:2,\r
+index f522f69eb933..2b54925bd5d1 100644\r
+--- a/test/corpus/cur/18:2,\r
++++ b/test/corpus/cur/18:2,\r
+@@ -3,6 +3,7 @@ To: notmuch@notmuchmail.org\r
+ Date: Tue, 17 Nov 2009 18:21:38 -0500\r
+ Subject: [notmuch] archive\r
+ Message-ID: <20091117232137.GA7669@griffis1.net>\r
++List-Id: <test1.example.com>\r
+ \r
+ Just subscribed, I'd like to catch up on the previous postings,\r
+ but the archive link seems to be bogus?\r
+diff --git a/test/corpus/cur/51:2, b/test/corpus/cur/51:2,\r
+index f522f69eb933..b155e6ee64a5 100644\r
+--- a/test/corpus/cur/51:2,\r
++++ b/test/corpus/cur/51:2,\r
+@@ -3,6 +3,7 @@ To: notmuch@notmuchmail.org\r
+ Date: Tue, 17 Nov 2009 18:21:38 -0500\r
+ Subject: [notmuch] archive\r
+ Message-ID: <20091117232137.GA7669@griffis1.net>\r
++List-Id: <test2.example.com>\r
+ \r
+ Just subscribed, I'd like to catch up on the previous postings,\r
+ but the archive link seems to be bogus?\r
+diff --git a/test/search b/test/search\r
+index a7a0b18d2e48..bef42971226c 100755\r
+--- a/test/search\r
++++ b/test/search\r
+@@ -129,4 +129,28 @@ add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12\r
+ output=$(notmuch search "bödý" | notmuch_search_sanitize)\r
+ test_expect_equal "$output" "thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"\r
+ \r
++test_begin_subtest "Search by List-Id"\r
++notmuch search list:notmuch.notmuchmail.org | notmuch_search_sanitize > OUTPUT\r
++cat <<EOF >EXPECTED\r
++thread:XXX 2009-11-18 [2/2] Lars Kellogg-Stedman; [notmuch] "notmuch help" outputs to stderr? (attachment inbox signed unread)\r
++thread:XXX 2009-11-18 [4/7] Lars Kellogg-Stedman, Mikhail Gusarov| Keith Packard, Carl Worth; [notmuch] Working with Maildir storage? (inbox signed unread)\r
++thread:XXX 2009-11-18 [1/2] Alex Botero-Lowry| Carl Worth; [notmuch] [PATCH] Error out if no query is supplied to search instead of going into an infinite loop (attachment inbox unread)\r
++thread:XXX 2009-11-17 [1/3] Adrian Perez de Castro| Keith Packard, Carl Worth; [notmuch] Introducing myself (inbox signed unread)\r
++thread:XXX 2009-11-17 [1/2] Alex Botero-Lowry| Carl Worth; [notmuch] preliminary FreeBSD support (attachment inbox unread)\r
++EOF\r
++test_expect_equal_file OUTPUT EXPECTED\r
++\r
++test_begin_subtest "Search by List-Id, duplicated messages, step 1"\r
++notmuch search list:test1.example.com | notmuch_search_sanitize > OUTPUT\r
++cat <<EOF >EXPECTED\r
++thread:XXX 2009-11-17 [1/3] Aron Griffis| Keith Packard, Carl Worth; [notmuch] archive (inbox unread)\r
++EOF\r
++test_expect_equal_file OUTPUT EXPECTED\r
++\r
++test_begin_subtest "Search by List-Id, duplicated messages, step 2"\r
++notmuch search list:test2.example.com | notmuch_search_sanitize > OUTPUT\r
++cat <<EOF >EXPECTED\r
++thread:XXX 2009-11-17 [1/3] Aron Griffis| Keith Packard, Carl Worth; [notmuch] archive (inbox unread)\r
++EOF\r
++test_expect_equal_file OUTPUT EXPECTED\r
+ test_done\r
+diff --git a/test/test-lib.sh b/test/test-lib.sh\r
+index d8e0d9115a69..981bde4a4004 100644\r
+--- a/test/test-lib.sh\r
++++ b/test/test-lib.sh\r
+@@ -576,9 +576,9 @@ test_expect_equal_json () {\r
+ # The test suite forces LC_ALL=C, but this causes Python 3 to\r
+ # decode stdin as ASCII. We need to read JSON in UTF-8, so\r
+ # override Python's stdio encoding defaults.\r
+- output=$(echo "$1" | PYTHONIOENCODING=utf-8 python -mjson.tool \\r
++ output=$(echo "$1" | PYTHONIOENCODING=utf-8 python2 -mjson.tool \\r
+ || echo "$1")\r
+- expected=$(echo "$2" | PYTHONIOENCODING=utf-8 python -mjson.tool \\r
++ expected=$(echo "$2" | PYTHONIOENCODING=utf-8 python2 -mjson.tool \\r
+ || echo "$2")\r
+ shift 2\r
+ test_expect_equal "$output" "$expected" "$@"\r
+-- \r
+ Kirill A. Shutemov\r
projects / notmuch-archives.git / commitdiff