From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
To: Carl Worth <cworth@cworth.org>, notmuch@notmuchmail.org
Subject: Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
In-Reply-To: <87hb798uz7.fsf@yoom.home.cworth.org>
References: <1309236311-2162-1-git-send-email-dmitry.kurochkin@gmail.com>
	<1309236311-2162-10-git-send-email-dmitry.kurochkin@gmail.com>
	<87hb798uz7.fsf@yoom.home.cworth.org>
User-Agent: Notmuch/0.5-289-ge053759 (http://notmuchmail.org) Emacs/23.3.1
	(x86_64-pc-linux-gnu)
Date: Wed, 29 Jun 2011 02:22:31 +0400
Message-ID: <87wrg5wq8o.fsf@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Precedence: list

On Tue, 28 Jun 2011 15:14:04 -0700, Carl Worth <cworth@cworth.org> wrote:
> On Tue, 28 Jun 2011 08:45:11 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> > Before the change, every Emacs test ran in a separate Emacs
> > instance.  Starting Emacs many times wastes considerable time and
> > it gets worse as the test suite grows.  The patch solves this by
> > using a single Emacs server and emacsclient(1) to run multiple
> > tests. 
> 
> Great, great stuff, Dmitry!
> 
> I've pushed everything earlier than this patch in this series.
> 
> And I'd be fine pushing this one as well, (if Austin cares strongly
> about not polling, I'll invite him to improve things from here).
> 
> Except that things don't actually work for me with this patch applied.
> 
> I'm not getting consistent results from the test suite, (I have seen
> both "2 tests failed" and "5 tests failed"). Here are the failures from
> a recent run with 5 failures:
> 
>  FAIL   decryption, --format=text
>  FAIL   decryption, --format=json
>  FAIL   decryption, --format=json, --part=4
>  FAIL   decrypt attachment (--part=5 --format=raw)
>  FAIL   decryption failure with missing key
> 
> In each case the actual output was either empty, or an empty JSON
> array. In some cases notmuch also gave an error:
> 
> Error: search term did not match precisely one message.
> 
> Are you not getting these same failures?
> 

Ouch.  I never saw these.  Do you get these in crypto tests only?

Regards,
  Dmitry

> Let me know what else I can do to investigate this.
> 
> -Carl
> 
> -- 
> carl.d.worth@intel.com