From: Thomas Schwinge <thomas@schwinge.name>
To: notmuch@notmuchmail.org
Subject: Re: [BUG] Emacs UI dropping every 25th line, roughly
In-Reply-To: <87tygqm7g4.fsf@kepler.schwinge.homeip.net>
References: <87tygqm7g4.fsf@kepler.schwinge.homeip.net>
User-Agent: Notmuch/0.5-33-g665f77b (http://notmuchmail.org) Emacs/23.2.1
	(i486-pc-linux-gnu)
Date: Wed, 02 Feb 2011 17:12:16 +0100
Message-ID: <87zkqeiffj.fsf@kepler.schwinge.homeip.net>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"
Precedence: list

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hallo!

On Sun, 30 Jan 2011 22:02:03 +0100, I wrote:
> That is, roughly every 25th line is dropped in the Emacs notmuch-search
> buffer!  (Via setting notmuch-command to a shell script, I intercepted
> the notmuch / emacs pipeline with tee, and found that what comes out of
> ``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
> notmuch elisp code issue.)
>=20
> And, 25 times the medium length of a ``notmuch search [...]'' output line
> is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
> notmuch elisp code, or is Emacs doing something awful with (wild
> speculation...) short reads on buffer page boundaries (or whatever else)?

I began to analyze this.  Here is a dump of my steps.

I used strace on the (already running) Emacs process.  strace would log
to several files: one for the main (Emacs) process, and a new one for
each forked process.  I can see that a ``notmuch search [...]'' process
is fork()ed / exec()uted.  It does a lot of read()ing from the DB, and
write()s out the results record by record (which in our case means line
by line).  Up to the point where I examined, there have been no short
writes.  The log:

    [some write()s for the first two dozen search results]
    16:26:54.143464 write(1, "thread:00000000000006c7   2009-12-16 [2/2] Th=
omas Schwinge; [subject] ([flags])\n", 135) =3D 135
    [...]
    16:26:54.266866 write(1, "thread:0000000000000ac4   2009-12-16 [1/1] js=
m28@sourceware.org; [subject] ([flags])\n", 181) =3D 181

At this / in the middle of this point, the 4 KiB size is hit.  After
accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
first chunk (line breaks inserted for clarity):

    16:26:57.928798 read(8, "[first two dozen results]"
                            "thread:00000000000006c7   2009-12-16 [2/2] Tho=
mas Schwinge; [subject] ([flags])\n"
                            "t", 4096) =3D 4095

The last result line is obviously incomplete, only the `t' so far.

There is more to be read, so Emacs quickly goes on with the next chunk:

    16:26:57.966247 read(8, "hread:0000000000000ac4   2009-12-16 [1/1] jsm2=
8@sourceware.org; [subject] ([flags])\n"
                            "[more results]", 4096) =3D 4095

That's the remainder of the partial line, plus further results.  This is
all fine.  It's now Emacs' job to re-assemble the two partial lines
(thread ac4).  I note that indeed this search result is missing in the
notmuch-search buffer.

(By the way, I wonder why read() returns 4095 bytes instead of 4096?)

I'll next move on to debugging what actually appears at the elisp layer.
I somehow don't think the Emacs core is faulty; probably rather our
asynchronous results presentation layer handles partial lines
incorrectly?  More to come later.


Gr=C3=BC=C3=9Fe,
 Thomas

--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQEcBAEBAgAGBQJNSYJgAAoJEGe3hdm9kOiiqm4H/ReVsJ9ND/UH+IMKkvMvyzsJ
DJGg8hjrzfmFJmIKyC8I0+JHtHBGQtExgEy4GfkTIKkt1/7n/fVm0JVOG0p3yJY8
mdagH7Cf3nI34zPn5vfZjtD0rn8EaezKbus67oO+3NGbjwgXEwg6K6RUdQSOhOq/
V8giOpUwfjf48U2xVsg6+09jQnRf51LylbC2NoKEQyc/shDmsHV2rbrf/Ml6Plhp
/O8AvhWNHoYx4LpUDvPY4U0bjEp5/rPPPIOJY/VcgDXRztpMABoGJsnQnu684k06
kzWi8yn7BHzjL9ZjSclVm8/J/v168nW1TtOJ5nxKtfaSYP4l7J0AX1ENEiZV8Xs=
=BH3d
-----END PGP SIGNATURE-----
--=-=-=--