From: Jani Nikula Date: Sun, 28 Oct 2012 22:39:02 +0000 (+0200) Subject: Re: [PATCH v5 6/9] lib: add date range query support X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=6e012eddc04e0808e0ab62bbd59be95ce1e40be9;p=notmuch-archives.git Re: [PATCH v5 6/9] lib: add date range query support --- diff --git a/18/9e1728ee51c56e9dd1f7eb9e7158f84c9bc2a8 b/18/9e1728ee51c56e9dd1f7eb9e7158f84c9bc2a8 new file mode 100644 index 000000000..f00245aae --- /dev/null +++ b/18/9e1728ee51c56e9dd1f7eb9e7158f84c9bc2a8 @@ -0,0 +1,271 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id D1479431FBC + for ; Sun, 28 Oct 2012 15:39:08 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id yeHPd1hON8rc for ; + Sun, 28 Oct 2012 15:39:08 -0700 (PDT) +Received: from mail-la0-f53.google.com (mail-la0-f53.google.com + [209.85.215.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id A29E0431FAF + for ; Sun, 28 Oct 2012 15:39:07 -0700 (PDT) +Received: by mail-la0-f53.google.com with SMTP id l5so3914390lah.26 + for ; Sun, 28 Oct 2012 15:39:06 -0700 (PDT) +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=google.com; s=20120113; + h=from:to:cc:subject:in-reply-to:references:user-agent:date + :message-id:mime-version:content-type:x-gm-message-state; + bh=ScbGARoCtCCJH8kzhb6Irqez/Pu58HowWEKT5iQ8kB8=; + b=PefvjM5xS5Cj9yP6frqaNmN0DH7CX8sTpkgAAg2PyF3YevTS5MxV1ocF55jZ0Q4pMS + /uamChACfFm6xvX7tCbLzlAsvjq+irioVKbTUBIRU/NkmnSNzKA4Nu66rNLmifhNg6UZ + M6YbNRDTOz+XpGmh25WT3wrxfNCXq3SXxCF8VxOspqiDYhP/Oj9/XmSliZFtDSZ4jywj + 3Zqa2cm+B2wGUZqULN5MV0k/7v0hANn5jTdBfEnWyjvf+6p1xaVJn0wM3azRT5yPOHJP + cJWWapUdPusC50Jg7zDffbENn4nCFk/XpR2LA01wgAWrC+FOpUwJXYuifaGKKvOrHxIA + KaxA== +Received: by 10.112.38.234 with SMTP id j10mr10886774lbk.80.1351463946143; + Sun, 28 Oct 2012 15:39:06 -0700 (PDT) +Received: from localhost (dsl-hkibrasgw4-fe51df00-27.dhcp.inet.fi. + [80.223.81.27]) + by mx.google.com with ESMTPS id b8sm2560120lbn.8.2012.10.28.15.39.04 + (version=SSLv3 cipher=OTHER); Sun, 28 Oct 2012 15:39:05 -0700 (PDT) +From: Jani Nikula +To: Austin Clements +Subject: Re: [PATCH v5 6/9] lib: add date range query support +In-Reply-To: <20121023045255.GQ14861@mit.edu> +References: + + <20121023045255.GQ14861@mit.edu> +User-Agent: Notmuch/0.14+46~g272a1f1 (http://notmuchmail.org) Emacs/23.4.1 + (i686-pc-linux-gnu) +Date: Mon, 29 Oct 2012 00:39:02 +0200 +Message-ID: <87fw4ykyrd.fsf@nikula.org> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-Gm-Message-State: + ALoCoQnSr8KJpnBqalsJwDBV/FlpKM0eGIIBtz+v9ewruXJx+61/oBqNR14Jtf+rmSugqaFS5lZF +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 28 Oct 2012 22:39:09 -0000 + +On Tue, 23 Oct 2012, Austin Clements wrote: +> Quoth Jani Nikula on Oct 22 at 12:22 am: +>> Add a custom value range processor to enable date and time searches of +>> the form date:since..until, where "since" and "until" are expressions +>> understood by the previously added date/time parser, to restrict the +>> results to messages within a particular time range (based on the Date: +>> header). +>> +>> If "since" or "until" describes date/time at an accuracy of days or +>> less, the values are rounded according to the accuracy, towards past +>> for "since" and towards future for "until". For example, +>> date:november..yesterday would match from the beginning of November +>> until the end of yesterday. Expressions such as date:today..today +>> means since the beginning of today until the end of today. +>> +>> Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can +>> specify date:..until or date:since.. to not limit the start or end +>> date, respectively. +>> +>> CAVEATS: +>> +>> Xapian does not support spaces in range expressions. You can replace +>> the spaces with '_', or (in most cases) '-', or (in some cases) leave +>> the spaces out altogether. +>> +>> Entering date:expr without ".." (for example date:yesterday) will not +>> work as you might expect. You can achieve the expected result by +>> duplicating the expr both sides of ".." (for example +>> date:yesterday..yesterday). +>> +>> Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't +>> produce an error either. +>> +>> Signed-off-by: Jani Nikula +>> --- +>> lib/Makefile.local | 1 + +>> lib/database-private.h | 1 + +>> lib/database.cc | 5 +++++ +>> lib/parse-time-vrp.cc | 40 ++++++++++++++++++++++++++++++++++++++++ +>> lib/parse-time-vrp.h | 19 +++++++++++++++++++ +>> 5 files changed, 66 insertions(+) +>> create mode 100644 lib/parse-time-vrp.cc +>> create mode 100644 lib/parse-time-vrp.h +>> +>> diff --git a/lib/Makefile.local b/lib/Makefile.local +>> index d1635cf..6c0f42f 100644 +>> --- a/lib/Makefile.local +>> +++ b/lib/Makefile.local +>> @@ -58,6 +58,7 @@ libnotmuch_c_srcs = \ +>> +>> libnotmuch_cxx_srcs = \ +>> $(dir)/database.cc \ +>> + $(dir)/parse-time-vrp.cc \ +>> $(dir)/directory.cc \ +>> $(dir)/index.cc \ +>> $(dir)/message.cc \ +>> diff --git a/lib/database-private.h b/lib/database-private.h +>> index 88532d5..d3e65fd 100644 +>> --- a/lib/database-private.h +>> +++ b/lib/database-private.h +>> @@ -52,6 +52,7 @@ struct _notmuch_database { +>> Xapian::QueryParser *query_parser; +>> Xapian::TermGenerator *term_gen; +>> Xapian::ValueRangeProcessor *value_range_processor; +>> + Xapian::ValueRangeProcessor *date_range_processor; +>> }; +>> +>> /* Return the list of terms from the given iterator matching a prefix. +>> diff --git a/lib/database.cc b/lib/database.cc +>> index 761dc1a..4df3217 100644 +>> --- a/lib/database.cc +>> +++ b/lib/database.cc +>> @@ -19,6 +19,7 @@ +>> */ +>> +>> #include "database-private.h" +>> +#include "parse-time-vrp.h" +>> +>> #include +>> +>> @@ -710,12 +711,14 @@ notmuch_database_open (const char *path, +>> notmuch->term_gen = new Xapian::TermGenerator; +>> notmuch->term_gen->set_stemmer (Xapian::Stem ("english")); +>> notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP); +>> + notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP); +>> +>> notmuch->query_parser->set_default_op (Xapian::Query::OP_AND); +>> notmuch->query_parser->set_database (*notmuch->xapian_db); +>> notmuch->query_parser->set_stemmer (Xapian::Stem ("english")); +>> notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME); +>> notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor); +>> + notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor); +>> +>> for (i = 0; i < ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) { +>> prefix_t *prefix = &BOOLEAN_PREFIX_EXTERNAL[i]; +>> @@ -778,6 +781,8 @@ notmuch_database_close (notmuch_database_t *notmuch) +>> notmuch->xapian_db = NULL; +>> delete notmuch->value_range_processor; +>> notmuch->value_range_processor = NULL; +>> + delete notmuch->date_range_processor; +>> + notmuch->date_range_processor = NULL; +>> } +>> +>> void +>> diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc +>> new file mode 100644 +>> index 0000000..7e4eca4 +>> --- /dev/null +>> +++ b/lib/parse-time-vrp.cc +>> @@ -0,0 +1,40 @@ +> +> Should this file have the usual preamble? + +Probably, yes. + +>> + +>> +#include "database-private.h" +>> +#include "parse-time-vrp.h" +>> +#include "parse-time-string.h" +>> + +>> +#define PREFIX "date:" +>> + +>> +/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */ +>> +Xapian::valueno +>> +ParseTimeValueRangeProcessor::operator() (std::string &begin, std::string &end) +>> +{ +>> + time_t t, now; +>> + +>> + /* Require date: prefix in start of the range... */ +>> + if (STRNCMP_LITERAL (begin.c_str (), PREFIX)) +> +> Could be +> if (begin.rfind (PREFIX, 0) == string::npos) +> but that may not be clearer. + +Not to me at least; my C++ is rusty. + +>> + return Xapian::BAD_VALUENO; +>> + +>> + /* ...and remove it. */ +>> + begin.erase (0, sizeof (PREFIX) - 1); +>> + +>> + /* Use the same 'now' for begin and end. */ +>> + if (time (&now) == (time_t) -1) +>> + return Xapian::BAD_VALUENO; +>> + +>> + if (!begin.empty ()) { +>> + if (parse_time_string (begin.c_str (), &t, &now, PARSE_TIME_ROUND_DOWN)) +>> + return Xapian::BAD_VALUENO; +>> + +>> + begin.assign (Xapian::sortable_serialise ((double) t)); +>> + } +>> + +>> + if (!end.empty ()) { +>> + if (parse_time_string (end.c_str (), &t, &now, PARSE_TIME_ROUND_UP_INCLUSIVE)) +>> + return Xapian::BAD_VALUENO; +>> + +>> + end.assign (Xapian::sortable_serialise ((double) t)); +>> + } +>> + +>> + return valno; +>> +} +>> diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h +>> new file mode 100644 +>> index 0000000..526c217 +>> --- /dev/null +>> +++ b/lib/parse-time-vrp.h +>> @@ -0,0 +1,19 @@ +> +> Same thing about the preamble. +> +>> + +>> +#ifndef NOTMUCH_PARSE_TIME_VRP_H +>> +#define NOTMUCH_PARSE_TIME_VRP_H +>> + +>> +#include +>> + +>> +/* see *ValueRangeProcessor in xapian-core/include/xapian/queryparser.h */ +> +> Out of curiosity, why the Xapian source reference? +> ValueRangeProcessor is documented along the rest of Xapian. + +To be honest, I couldn't write this with the documentation alone, and +Xapian has quite a bit of source code, so I wrote it down for me. I +figured it does no harm to leave it there. + +BR, +Jani. + +>> +class ParseTimeValueRangeProcessor : public Xapian::ValueRangeProcessor { +>> +protected: +>> + Xapian::valueno valno; +>> + +>> +public: +>> + ParseTimeValueRangeProcessor (Xapian::valueno slot_) +>> + : valno(slot_) { } +>> + +>> + Xapian::valueno operator() (std::string &begin, std::string &end); +>> +}; +>> + +>> +#endif /* NOTMUCH_PARSE_TIME_VRP_H */