From: Aaron Ecay <aaronecay@gmail.com>
To: David Edmondson <dme@dme.org>, Daniel Schoepe <daniel@schoepe.org>,
	James Vasile <james@hackervisions.org>, notmuch@notmuchmail.org
Subject: Re: [notmuch] [PATCH] Calls to notmuch get queued and executed
	asynchronously.
In-Reply-To: <cun4nwi8h7i.fsf@hotblack-desiato.hh.sledj.net>
References: <87vddnlxos.wl%james@hackervisions.org>
	<87wrbykr13.fsf@gilead.invalid>
	<cun4nwi8h7i.fsf@hotblack-desiato.hh.sledj.net>
User-Agent: Notmuch/0.10.1+56~gd709fd6 (http://notmuchmail.org)
	Emacs/24.0.92.3 (i386-apple-darwin10.8.0)
Date: Fri, 30 Dec 2011 14:21:31 -0500
Message-ID: <m2fwg1uapw.fsf@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list

On Fri, 30 Dec 2011 10:52:17 +0000, David Edmondson <dme@dme.org> wrote:
> > I discovered this patch a while ago and it almost applies cleanly (the
> > conflicts are easy to fix though) and it has made the emacs UI for
> > notmuch _much_ more responsive and enjoyable for me.

This is definitely true for me as well.  So much so that I reinvented
this patch from scratch a couple of weeks ago.  (As Picasso said, good
artists copy, great artists steal.)

> > As discussed on IRC, issues such as where to handle retries in case
> > the database is locked should probably be handled in the notmuch
> > binary instead of each UI.

Hmm.  When my implementation detects that it cannot get the DB lock
after trying several times (up to ~2 mins of waiting), it stops
trying but continues to queue up operations.  The user can then
manually restart the queue.  This isn=E2=80=99t a principled solution, but
it avoids losing tagging operations from emacs while a long-running
process has the DB locked.  (My original motivation for writing the
patch was actually that my new-mail script sometimes holds the DB
lock when I am trying to read mail, leading to emacs errors and
dropped tagging operations.  The speedup was a welcome side effect.)

I haven=E2=80=99t had a chance to carefully look at the patch you found, so=
 IDK
what its behavior would be in this case.

(The other thing that I dislike about the patch you found is that it
uses a call to the =E2=80=9Csleep=E2=80=9D command to wait, rather than usi=
ng built-in
emacs functionality.)

> It would be good to have an updated version of this patch if it is still
> considered useful, though I don't suffer particularly from lag in the
> emacs UI.

The emacs UI has small delays, which dropped away when I applied my
version of this patch.  After being conditioned to expect them, I was
surprised to find how much faster everything seemed.  My hardware is old
but not ancient (5.5 year old Macbook; I suspect the bottleneck for
notmuch is the 5400rpm HDD).  Depending on your setup, you too might be
pleasantly surprised.

The test suite is borked on OS X and I don=E2=80=99t have access to a linux
machine while traveling.  I was planning on sending my patch in early
Jan. when I had a chance to verify it under the test suite (probably
requiring some changes to the emacs test library to make it async-safe).
I also haven=E2=80=99t tested the patch on Emacs versions older than 24 =E2=
=80=93 but I
don=E2=80=99t think there are any impediments to compatibility with v.23 (n=
ot
sure about earlier versions).

Since there is interest, I=E2=80=99ll go ahead and send it now with all the
usual caveats about code under development.  I have been using the patch
for a couple of weeks without problems, though.  Daniel, if you want to
un-conflict (and squash) the patches from James that might be useful, at
least to compare the two approaches.

Aaron

-----cut-here-----

>From f0a0fe04254d9b5e17c873b293c6a5a270cb909a Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Mon, 19 Dec 2011 12:14:31 -0500
Subject: [PATCH] [emacs] add async tagging support

still a WIP
---
 emacs/notmuch.el |   55 ++++++++++++++++++++++++++++++++++++++++++++++++++=
+++-
 1 files changed, 54 insertions(+), 1 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index fde2377..ca077c2 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -446,6 +446,59 @@ Complete list of currently available key bindings:
   (let ((message-id (notmuch-search-find-thread-id)))
     (notmuch-mua-new-reply message-id prompt-for-sender)))
=20
+(defvar notmuch-process-queue nil
+  "Queue of pending notmuch tag operations.
+
+Each entry in the queue is a list of strings, which are arguments
+to be passed to the notmuch command.")
+
+(defvar notmuch-process-current-command nil
+  "The currently executing notmuch command arguments")
+
+(defvar notmuch-process-wait-time 1.0
+  "How long to wait for the db lock, in seconds.")
+
+(defvar notmuch-process-error nil)
+
+(defun notmuch-process-restart-queue ()
+  (interactive)
+  (setq notmuch-process-error nil)
+  (notmuch-process-kick-queue))
+
+(defun notmuch-process-kick-queue ()
+  (when notmuch-process-error
+    ;; TODO: better error msg, tell how to manually kick
+    (error "Notmuch couldn't get the DB lock after trying for more than a =
minute."))
+  (when (and notmuch-process-queue
+	     (memq (process-status "notmuch") '(exit nil)))
+    (let ((args (car notmuch-process-queue))
+	  proc)
+      (setq notmuch-process-queue (cdr notmuch-process-queue)
+	    notmuch-process-current-command args
+	    proc (apply #'start-process "notmuch"
+			(get-buffer-create "*Notmuch output*")
+			notmuch-command args))
+      (set-process-sentinel proc #'notmuch-process-sentinel))))
+
+(defun notmuch-process-sentinel (proc status)
+  (if (equal status "finished\n")
+      (progn
+	(setq notmuch-process-wait-time 1.0
+	      notmuch-process-current-command nil)
+	(notmuch-process-kick-queue))
+    (setq notmuch-process-wait-time (* 2 notmuch-process-wait-time)
+	  notmuch-process-queue (cons notmuch-process-current-command
+				      notmuch-process-queue))
+    (if (<=3D notmuch-process-wait-time 64)
+	(progn
+	  (run-at-time notmuch-process-wait-time nil
+		       #'notmuch-process-kick-queue))
+      (setq notmuch-process-error t))))
+
+(defun notmuch-process-queue-command (&rest args)
+  (setq notmuch-process-queue (append notmuch-process-queue (list args)))
+  (notmuch-process-kick-queue))
+
 (defun notmuch-call-notmuch-process (&rest args)
   "Synchronously invoke \"notmuch\" with the given list of arguments.
=20
@@ -474,7 +527,7 @@ messages instead of running (notmuch-call-notmuch-proce=
ss \"tag\" ..)
 directly, so that hooks specified in notmuch-before-tag-hook and
 notmuch-after-tag-hook will be run."
   (run-hooks 'notmuch-before-tag-hook)
-  (apply 'notmuch-call-notmuch-process
+  (apply 'notmuch-process-queue-command
 	 (append (list "tag") tags (list "--" query)))
   (run-hooks 'notmuch-after-tag-hook))
=20
--=20
1.7.8.2

-----cut-here-----

--=20
Aaron Ecay