From: David Bremner <bremner@debian.org>
To: "Notmuch Mail" <notmuch@notmuchmail.org>
Subject: More ideas about logging.
User-Agent: Notmuch/0.10.2+100~g7a23455 (http://notmuchmail.org) Emacs/23.3.1
	(x86_64-pc-linux-gnu)
Date: Thu, 15 Dec 2011 22:09:08 -0400
Message-ID: <87obv9i7y3.fsf@zancas.localnet>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
	micalg=pgp-sha1; protocol="application/pgp-signature"
Cc: Olly Betts <olly@survex.com>
Precedence: list

--=-=-=
Content-Transfer-Encoding: quoted-printable


Various discussions (mostly on IRC) from my jlog proposal, and a from
Thomas's mtime
(id:"1323796305-28789-1-git-send-email-schnouki@schnouki.net") proposal
got me thinking.  So let me know what you think about the following.

The goal here is to log tag adds and deletes (including those implicit
in message deletion) to facilitate tag synchonization.

If we use Xapian to store transaction numbers (much as the
last_thread_id is stored now), then we don't need an external logging
library. We can rely on the xapian to keep other clients from writing

Assume we have routines read_metadata and write_metadata that read and
write to the xapian database metadata (in real life, I think we might
need to decide in advance exactly what will be written there).

when we create a database

write_metadata('log_write',0)
write_metadata('log_read',0) // more about this later

To carry out database operation X with logging, we do the following

begin_atomic

    txn=3Dread_metadata('last_written')

    X

    // begin dangerzone
    fprintf(logfile,"%d %s",num+1,stuff) // or whatever.

    write_metadata('last_written', num+1)

end_atomic
//end dangerzone

If I understand correctly, then the only way the database and the log
can get out of sync is if this is interrupted in the "dangerzone"
between the start of the log write and the end of the xapian atomic
transaction. But then since we can consider the database authoritative
(since our goal is synchonization rather than recovery), we can discard
those portions of the log. We have to be a bit careful to discard
incomplete log items at the end of the log (maybe a checksum?).

So how do we discard? Two places. At the opening of the database for
writing, we truncate the log file (if we are very lazy, we can use seek
offsets as transaction indicies to facilitate this).=20

In order to guarantee that log item is output exactly once, it seems
like we need another counter (or maybe I'm overthinking this)

     read_ptr =3D read_metadata('last_read')

     write_ptr =3D read_metadata('last_write')
=20=20=20=20=20
     while (read_ptr < write_ptr) {
         begin_atomic
            s =3D read(read_ptr)
            do_stuff(s)
            read_ptr++
            write_metadata('log_read', read_ptr);
         end_atomic
     }

     write_metadata('log_write',0) // The log file will be truncated on
                                   // on db open
     write_metadata('log_read',0)=20

I think we can double check if write_ptr <=3D read_ptr on next db open,
and truncate then if needed.

I think we need to assume that do_stuff is atomic here; I'm not sure how
reasonable or unreasonable that is in practice.

I also don't know about the performance implications of reading and
writing like maniac from the xapian metadata. Of course if this whole
scheme is fatally flawed, no need to worry about performance.

I don't think the actual amount of code involved would be too bad. Of
course, I thought was going to be a short message too.

d


--=-=-=
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iJwEAQECAAYFAk7qqEUACgkQTiiN/0Um85mVgAQAop6yw5qniOmbqUn5XkmH0eTx
YdDwkopNIyHPfRzZP4ZJIkYyoTnvfK94fd8rhsM8CWSFfORhwRLRWbjLZb5iyTHV
Nk+8gAHiDCueoJ+lI01uxrrxWQmzLYz1qwK6e2zYtqp9rU2INbn8lOgaBDEV52Fq
y2oJBmVaUI8p9eVQSaM=
=cSaO
-----END PGP SIGNATURE-----
--=-=-=--