From 1d673c8b74f8a66f1e6bc3a3e7cabdcd46c57e98 Mon Sep 17 00:00:00 2001 From: Austin Clements Date: Tue, 13 Aug 2013 10:34:26 +2000 Subject: [PATCH] Re: Possible addtions to notmuch new ? --- cd/c2e668921beac42934bd9ac7b6063697ca5c90 | 153 ++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 cd/c2e668921beac42934bd9ac7b6063697ca5c90 diff --git a/cd/c2e668921beac42934bd9ac7b6063697ca5c90 b/cd/c2e668921beac42934bd9ac7b6063697ca5c90 new file mode 100644 index 000000000..1ab4cd253 --- /dev/null +++ b/cd/c2e668921beac42934bd9ac7b6063697ca5c90 @@ -0,0 +1,153 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 947BE431FBC + for ; Mon, 12 Aug 2013 07:34:41 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id jlc2fYe5+FOD for ; + Mon, 12 Aug 2013 07:34:35 -0700 (PDT) +Received: from dmz-mailsec-scanner-3.mit.edu (dmz-mailsec-scanner-3.mit.edu + [18.9.25.14]) + by olra.theworths.org (Postfix) with ESMTP id 4A18F431FAF + for ; Mon, 12 Aug 2013 07:34:35 -0700 (PDT) +X-AuditID: 1209190e-b7f988e0000009a7-80-5208f277a9ec +Received: from mailhub-auth-4.mit.edu ( [18.7.62.39]) + by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP + id 79.10.02471.872F8025; Mon, 12 Aug 2013 10:34:32 -0400 (EDT) +Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) + by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id r7CEYUoX006416; + Mon, 12 Aug 2013 10:34:31 -0400 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r7CEYSnq030359 + (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); + Mon, 12 Aug 2013 10:34:29 -0400 +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) + (envelope-from ) + id 1V8tCV-0006nu-KX; Mon, 12 Aug 2013 10:34:27 -0400 +Date: Mon, 12 Aug 2013 10:34:26 -0400 +From: Austin Clements +To: Vladimir Marek +Subject: Re: Possible addtions to notmuch new ? +Message-ID: <20130812143426.GA13257@mit.edu> +References: <20130812093443.GB16684@virt.cz.oracle.com> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <20130812093443.GB16684@virt.cz.oracle.com> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFupjleLIzCtJLcpLzFFi42IRYrdT1634xBFkMPkVq8X1mzOZLTpu72Zz + YPJ4tuoWs8fHp7dYApiiuGxSUnMyy1KL9O0SuDJ6Vk5kKWiXqDj5zruB8Z1QFyMnh4SAicSE + Q8+YIWwxiQv31rN1MXJxCAnsY5SY+PstlLORUWJazwk2kCohgdNMEr23IiESSxglpnQvBmtn + EVCV2DNzHVgRm4CGxLb9yxlBbBEBPYlNm46xgtjMAtIS3343M4HYwgL6Ejev9rKA2LwCOhLz + JrczQSwwlzi4cjkTRFxQ4uTMJywQvVoSN/69BIpzgM1Z/o8DJMwpYCExc1YD2CpRARWJKSe3 + sU1gFJqFpHsWku5ZCN0LGJlXMcqm5Fbp5iZm5hSnJusWJyfm5aUW6Rrr5WaW6KWmlG5iBAU1 + pyTfDsavB5UOMQpwMCrx8GZ+5AgSYk0sK67MPcQoycGkJMq78wNQiC8pP6UyI7E4I76oNCe1 + +BCjBAezkgiv2jygHG9KYmVValE+TEqag0VJnPfZ07OBQgLpiSWp2ampBalFMFkZDg4lCV4L + kD2CRanpqRVpmTklCGkmDk6Q4TxAw31AaniLCxJzizPTIfKnGHU5/qyc+4lRiCUvPy9VSpzX + A6RIAKQoozQPbg4sGb1iFAd6S5g3EKSKB5jI4Ca9AlrCBLTEqBlsSUkiQkqqgXHD2sOmllsk + 3mz3nGeSGeoed1mkomPelpUWqxfXfXDpmNV0VOiPfur0aul134Qkf3Yv2WnROq3lU/w11zfr + HCYKyOcWuszhOLNUsVlKT+xA9KSJBtsnfpKq+aFx6MLe93Ez5aS3iGaVvTx3oiRu43rr0uV/ + vAwUzzC9ucHQeN92odkUyykzoluUWIozEg21mIuKEwHNfNWjIQMAAA== +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 12 Aug 2013 14:34:41 -0000 + +Quoth Vladimir Marek on Aug 12 at 11:34 am: +> Hi, +> +> My mail setup is a directory containing several subdirectories each +> subdirectory corresponds to one real mail account I am using. Each mail +> account is synchronized differently - I am using offlineimap, fetchmeail +> or even synthetically created emails (I am writing very simple jabber<-> +> mail gate).Every now and then I am running 'notmuch new' to discover new +> emails and make them available in my MUA. +> +> That works pretty well, but has some disadvantages too +> - notmuch new takes very long time (30s) during which the notmuch +> database seems to be locked for any other updates from my MUA +> - notmuch new takes long time because it always processes my archive +> dir containing many files. That's mostly un-necessary as typically +> there's no new mail delivered + +Could you try this patch? It's basically untested other than passing +the test suite, though in principle the worst harm it could do is make +notmuch new miss new messages or think renames are deletions. If it +helps significantly with your performance problems, I'll clean it up +and add a test. + +diff --git a/notmuch-new.c b/notmuch-new.c +index faa33f1..196c5cb 100644 +--- a/notmuch-new.c ++++ b/notmuch-new.c +@@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch, + } + db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0; + ++ if (directory && db_mtime == fs_mtime && st.st_nlink == 2) ++ goto DONE; ++ + /* If the database knows about this directory, then we sort based + * on strcmp to match the database sorting. Otherwise, we can do + * inode-based sorting for faster filesystem operation. */ + + +> - I don't have the possibility of passing new mails through procmail. +> That would be useful for example for changing cron mail subjects, +> putting related automated mails into threads (bugzilla, etc.). +> +> +> I was thinking that if we could split the new mail discovery from +> it's processing, it would solve the disadvantages I'm facing. Something +> like +> +> notmuch new --verbose --dry-run [dir] | my_filter | notmuch insert - +> +> It would work +> - --dry-run would not lock and change the database +> - --verbose would print the changes to stdout/stderr. Something like: +> +> new mail/file.1 +> new mail/file.2 +> deleted mail/file.3 +> renamed mail/file.4 mail/file.5 +> ... +> +> [dir] would limit the scope of 'notmuch' new search to dir and it's +> subdirectories. Alternatively we could have set of include or exclude +> rules similarly to rsync (for example), but [dir] is easier to +> implement. +> +> 'my_filter' would be my script which could change the contents of emails +> before they are inserted into notmuch database. +> +> Notmuch insert would be able to not only add new mails, but also remove +> old ones or note that the file was renamed. +> +> How would this sound? +> +> I'm not saying I would implement this, I'm rather curious where would +> you like to see notmuch in the future. +> +> Cheers -- 2.26.2