Date: Thu, 15 Dec 2011 23:07:22 -0500
From: Austin Clements <amdragon@MIT.EDU>
To: David Bremner <bremner@debian.org>
Subject: Re: More ideas about logging.
Message-ID: <20111216040722.GC12245@mit.edu>
References: <87obv9i7y3.fsf@zancas.localnet>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87obv9i7y3.fsf@zancas.localnet>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: Olly Betts <olly@survex.com>, Notmuch Mail <notmuch@notmuchmail.org>
Precedence: list

Quoth David Bremner on Dec 15 at 10:09 pm:
> Assume we have routines read_metadata and write_metadata that read and
> write to the xapian database metadata (in real life, I think we might
> need to decide in advance exactly what will be written there).
> 
> when we create a database
> 
> write_metadata('log_write',0)
> write_metadata('log_read',0) // more about this later
> 
> To carry out database operation X with logging, we do the following
> 
> begin_atomic
> 
>     txn=read_metadata('last_written')
> 
>     X
> 
>     // begin dangerzone
>     fprintf(logfile,"%d %s",num+1,stuff) // or whatever.
> 
>     write_metadata('last_written', num+1)
> 
> end_atomic
> //end dangerzone

The trouble with this approach is that the OS doesn't have to flush
logfile to the disk platters in any particular order relative to the
updates to Xapian.  So, after someone trips over your plug, you could
come back with Xapian saying you have 500 log entries when your
logfile comes back with only 20.  The only way I know of to fix this
is to fsync after the logfile write, which would obviously have
performance issues.  But maybe there are cleverer ways?