c8/9c4db920e6e67eb32e602fac31232d721021c3

   1 Return-Path: <amdragon@mit.edu>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5         by olra.theworths.org (Postfix) with ESMTP id 7BF88431FD0\r
   6         for <notmuch@notmuchmail.org>; Thu, 15 Dec 2011 20:06:03 -0800 (PST)\r
   7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
   8 X-Spam-Flag: NO\r
   9 X-Spam-Score: -0.7\r
  10 X-Spam-Level: \r
  11 X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
  12         tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
  13 Received: from olra.theworths.org ([127.0.0.1])\r
  14         by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
  15         with ESMTP id W+w15xDJn+3b for <notmuch@notmuchmail.org>;\r
  16         Thu, 15 Dec 2011 20:06:01 -0800 (PST)\r
  17 Received: from dmz-mailsec-scanner-2.mit.edu (DMZ-MAILSEC-SCANNER-2.MIT.EDU\r
  18         [18.9.25.13])\r
  19         by olra.theworths.org (Postfix) with ESMTP id 8CF20429E26\r
  20         for <notmuch@notmuchmail.org>; Thu, 15 Dec 2011 20:06:01 -0800 (PST)\r
  21 X-AuditID: 1209190d-b7f576d0000008c4-83-4eeac3a8bf45\r
  22 Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
  23         by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP\r
  24         id B5.1E.02244.8A3CAEE4; Thu, 15 Dec 2011 23:06:00 -0500 (EST)\r
  25 Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
  26         by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id pBG45xYl018116; \r
  27         Thu, 15 Dec 2011 23:06:00 -0500\r
  28 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
  29         (authenticated bits=0)\r
  30         (User authenticated as amdragon@ATHENA.MIT.EDU)\r
  31         by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id pBG45vTh003354\r
  32         (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
  33         Thu, 15 Dec 2011 23:05:58 -0500 (EST)\r
  34 Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
  35         (envelope-from <amdragon@mit.edu>)\r
  36         id 1RbP4s-0003jJ-AX; Thu, 15 Dec 2011 23:07:22 -0500\r
  37 Date: Thu, 15 Dec 2011 23:07:22 -0500\r
  38 From: Austin Clements <amdragon@MIT.EDU>\r
  39 To: David Bremner <bremner@debian.org>\r
  40 Subject: Re: More ideas about logging.\r
  41 Message-ID: <20111216040722.GC12245@mit.edu>\r
  42 References: <87obv9i7y3.fsf@zancas.localnet>\r
  43 MIME-Version: 1.0\r
  44 Content-Type: text/plain; charset=us-ascii\r
  45 Content-Disposition: inline\r
  46 In-Reply-To: <87obv9i7y3.fsf@zancas.localnet>\r
  47 User-Agent: Mutt/1.5.21 (2010-09-15)\r
  48 X-Brightmail-Tracker:\r
  49  H4sIAAAAAAAAA+NgFuphleLIzCtJLcpLzFFi42IR4hTV1l1x+JWfQfNMS4uNy34yW1y/OZPZ\r
  50         4v7y96wOzB6/2uYyezxbdYvZY93OP+wBzFFcNimpOZllqUX6dglcGT3TZzAVLOCsOHXvLWMD\r
  51         4wL2LkZODgkBE4lP64+yQNhiEhfurWfrYuTiEBLYxygx8WA/WJGQwAZGiTtreSESJ5kk3q6d\r
  52         zAzhLGGUuPp+J1ALBweLgKrEv8OWIA1sAhoS2/YvZwSxRQTUJO4uawezmQU8JKYdOccEYgsD\r
  53         xVdd6WEFsXkFdCS6zv9ihVimI3FsZgcjRFxQ4uTMJywQvVoSN/69ZAJZxSwgLbH8HwdImFNA\r
  54         V2Ld271gd4oKqEhMObmNbQKj0Cwk3bOQdM9C6F7AyLyKUTYlt0o3NzEzpzg1Wbc4OTEvL7VI\r
  55         10gvN7NELzWldBMjONAleXcwvjuodIhRgINRiYc3wOqVnxBrYllxZe4hRkkOJiVR3pp9QCG+\r
  56         pPyUyozE4oz4otKc1OJDjBIczEoivOL2QDnelMTKqtSifJiUNAeLkjhvza6HfkIC6Yklqdmp\r
  57         qQWpRTBZGQ4OJQne94eAGgWLUtNTK9Iyc0oQ0kwcnCDDeYCGsx4GGV5ckJhbnJkOkT/FqCgl\r
  58         zvsHpFkAJJFRmgfXC0tErxjFgV4R5mUHaecBJjG47ldAg5mABm8PewEyuCQRISXVwMhzLe3S\r
  59         0/3STecSd02a8qqj/NqWIwd9VLRXFv/h8d2/ca5lu4BY8G623UJHt5/cYVe7qeb/ir9cJ5n1\r
  60         1y/Nsjx6+9vE+HPc6QU96o9d74WeOnz02Tr56PxHNp3C+t77t92wVDguteMCzy+tWF7Jk4c4\r
  61         k9jMuEudvJgmLf543F5n4qkfd46dPOalxFKckWioxVxUnAgAagFDfB8DAAA=\r
  62 Cc: Olly Betts <olly@survex.com>, Notmuch Mail <notmuch@notmuchmail.org>\r
  63 X-BeenThere: notmuch@notmuchmail.org\r
  64 X-Mailman-Version: 2.1.13\r
  65 Precedence: list\r
  66 List-Id: "Use and development of the notmuch mail system."\r
  67         <notmuch.notmuchmail.org>\r
  68 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
  69         <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  70 List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
  71 List-Post: <mailto:notmuch@notmuchmail.org>\r
  72 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  73 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
  74         <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  75 X-List-Received-Date: Fri, 16 Dec 2011 04:06:03 -0000\r
  76 \r
  77 Quoth David Bremner on Dec 15 at 10:09 pm:\r
  78 > Assume we have routines read_metadata and write_metadata that read and\r
  79 > write to the xapian database metadata (in real life, I think we might\r
  80 > need to decide in advance exactly what will be written there).\r
  81 > \r
  82 > when we create a database\r
  83 > \r
  84 > write_metadata('log_write',0)\r
  85 > write_metadata('log_read',0) // more about this later\r
  86 > \r
  87 > To carry out database operation X with logging, we do the following\r
  88 > \r
  89 > begin_atomic\r
  90 > \r
  91 >     txn=read_metadata('last_written')\r
  92 > \r
  93 >     X\r
  94 > \r
  95 >     // begin dangerzone\r
  96 >     fprintf(logfile,"%d %s",num+1,stuff) // or whatever.\r
  97 > \r
  98 >     write_metadata('last_written', num+1)\r
  99 > \r
 100 > end_atomic\r
 101 > //end dangerzone\r
 102 \r
 103 The trouble with this approach is that the OS doesn't have to flush\r
 104 logfile to the disk platters in any particular order relative to the\r
 105 updates to Xapian.  So, after someone trips over your plug, you could\r
 106 come back with Xapian saying you have 500 log entries when your\r
 107 logfile comes back with only 20.  The only way I know of to fix this\r
 108 is to fsync after the logfile write, which would obviously have\r
 109 performance issues.  But maybe there are cleverer ways?\r