From 9a894ca3926262e84eea79804fbf79ad0fc8b375 Mon Sep 17 00:00:00 2001 From: Mark Walters Date: Thu, 28 Jun 2012 21:45:17 +0100 Subject: [PATCH] Re: [RFC PATCH 00/14] modular mail stores based on URIs --- d7/37796577b4d5daf5f338fd5c55921ee38ace39 | 181 ++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 d7/37796577b4d5daf5f338fd5c55921ee38ace39 diff --git a/d7/37796577b4d5daf5f338fd5c55921ee38ace39 b/d7/37796577b4d5daf5f338fd5c55921ee38ace39 new file mode 100644 index 000000000..4464db26c --- /dev/null +++ b/d7/37796577b4d5daf5f338fd5c55921ee38ace39 @@ -0,0 +1,181 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 476BC431FB6 + for ; Thu, 28 Jun 2012 13:45:34 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -1.098 +X-Spam-Level: +X-Spam-Status: No, score=-1.098 tagged_above=-999 required=5 + tests=[DKIM_ADSP_CUSTOM_MED=0.001, FREEMAIL_FROM=0.001, + NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id wqxEs8BdWwyg for ; + Thu, 28 Jun 2012 13:45:33 -0700 (PDT) +Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 6E9B6431FAF + for ; Thu, 28 Jun 2012 13:45:33 -0700 (PDT) +Received: from smtp.qmul.ac.uk ([138.37.6.40]) + by mail2.qmul.ac.uk with esmtp (Exim 4.71) + (envelope-from ) + id 1SkLae-0001AU-EN; Thu, 28 Jun 2012 21:45:29 +0100 +Received: from 94-192-233-223.zone6.bethere.co.uk ([94.192.233.223] + helo=localhost) + by smtp.qmul.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) + (envelope-from ) + id 1SkLae-0004H0-1Z; Thu, 28 Jun 2012 21:45:24 +0100 +From: Mark Walters +To: Ethan +Subject: Re: [RFC PATCH 00/14] modular mail stores based on URIs +In-Reply-To: + +References: <1340656899-5644-1-git-send-email-ethan@betacantrips.com> + <877gutnmf1.fsf@qmul.ac.uk> + +User-Agent: Notmuch/0.13.2+63~g548a9bf (http://notmuchmail.org) Emacs/23.4.1 + (x86_64-pc-linux-gnu) +Date: Thu, 28 Jun 2012 21:45:17 +0100 +Message-ID: <87k3yrmahu.fsf@qmul.ac.uk> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-Sender-Host-Address: 94.192.233.223 +X-QM-SPAM-Info: Sender has good ham record. :) +X-QM-Body-MD5: 8e222e9d9c98afd4c482bb64b2c82a6f (of first 20000 bytes) +X-SpamAssassin-Score: -1.8 +X-SpamAssassin-SpamBar: - +X-SpamAssassin-Report: The QM spam filters have analysed this message to + determine if it is + spam. We require at least 5.0 points to mark a message as spam. + This message scored -1.8 points. + Summary of the scoring: + * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, + * medium trust + * [138.37.6.40 listed in list.dnswl.org] + * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail + provider * (markwalters1009[at]gmail.com) + * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay + * domain + * 0.5 AWL AWL: From: address is in the auto white-list +X-QM-Scan-Virus: ClamAV says the message is clean +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 28 Jun 2012 20:45:34 -0000 + +On Thu, 28 Jun 2012, Ethan wrote: +> I sent this at first as a reply-only-to-sender. Oops! Sorry Mark for the +> double send. +> +> On Wed, Jun 27, 2012 at 5:17 AM, Mark Walters wrote: +> +>> > Personally, this isn't my favorite approach, for the following reasons: +>> > +>> > 1. Notmuch, at some point in its history, chose to store file paths +>> > relative to a "mail database", with the intent that if this mail +>> > database was moved, filenames would not change and everything would +>> > Just Work (tm). The above scheme completely reverses this design +>> > decision, and in general completely breaks this relocatability. I +>> > don't see any easy way to handle this problem. This isn't just a +>> > wishlist feature; at least two things in the test suite (caching of +>> > corpus.mail, and the atomicity tests) rely on this behavior. +>> +>> Why can't the URI just store a relative path, at least for maildir:// +>> and mbox:// ? It is purely internal to notmuch so it doesn't need to be +>> very standard. +>> +> +> Well, relative to where? This is especially relevant now that we can have +> multiple mail stores. It sounds like you are suggesting that all mbox:// +> URIs are relative to an "mbox root", but the fundamental question is how to +> pass that information from the configuration into the library. + +I was thinking of just having one mail root and inside that there could +be maildirs and mboxes. Everything would still be relative to the root. + +> Even using configuration itself may be problematic, because only the CLI +> uses the configuration, and language bindings like Python and Ruby might +> get out of sync! (But note also that the Python bindings currently use +> .notmuch-config to find the database path, so maybe it's not a big deal.) +> +> If I could do whatever I wanted, every mailstore would get registered +> somehow and the URIs could use those registered names to specify what +> they're relative to: maybe using hostname, such as +> maildir://university-mail/some-mail-file, mbox://old-unix-system/some.mbox. +> Then changing these names in .notmuch-config would be fine. I just don't +> know how to pass that configuration information without an approach like in +> the past patch series. +> +> > 2. Mail access information, i.e. open connections, etc. can only be +>> > stored in variables global to the mailstore code, and cannot be stored +>> > as private members of a mailstore object. This is more an aesthetic +>> > concern than a functional one. +>> > +>> > Anyhow, the following (enormous) patch series implement this design. I +>> > used uriparser as an external library to parse URIs. The API for this +>> > library is a little idiosyncratic. uriparser supports parsing Unicode +>> > URIs (strings of wchar_t), but I just used ASCII filenames because I +>> > think that's what comes out of Xapian. +>> +>> Why use a library? Isn't it just a question of does the string contain +>> // and, if so, splitting it? I guess that // is a nice separator as I +>> think we can assume that a true path does not contain it (since a +>> filename cannot contain /). +>> +> +> The URIs are true URIs. Filenames are provided by the "path" segment of the +> uri -- everything from the first slash after the hostname up to a ? for +> query arguments. My concern was that filenames could (in theory) contain # +> or ?, and in practice they contain : (maildir flags). I figured it was +> better to do it right. + +This is similar to your question to Jamie: + +> 1. Are URIs the way to specify individual messages, despite bremner's +> concerns about too much of the API being strings? Is adding another library +> is the easiest way to parse URIs? + +In my opinion the nice thing about using strings is that it does not require +any changes to the Xapian database to store them. I think using URIs may +not be best though as they seem to be annoying to parse (as filenames +can contain the same characters) and you seem to need to work around the +parser in some cases. + +I wonder if the following would be practical: use // as the field +separator: + +e.g. mbox://filename//start_of_message+length + +I think 2 consecutive slashes // is about the only thing we can assume +is not in the path or filename. Since it is not in the filename I think +parsing should be trivial (thus avoiding the extra library). + +Secondly, I would prefer to keep maildirs as just the bare file name: so +the existence of // can be the signal that there is some other +scheme. This is asymmetric, but is rather more backwardly compatible. + +I have read most of the patches and will send a couple of specific +comments but I completely agree with you that the first thing is to +decide the above. + +Finally do note these are just my views and others may have very +different ideas! + +Best wishes + +Mark + -- 2.26.2