--- /dev/null
+Return-Path: <amdragon@mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 2C0BD431FAF\r
+ for <notmuch@notmuchmail.org>; Fri, 8 Aug 2014 11:19:11 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id BBm2OGBfSuAc for <notmuch@notmuchmail.org>;\r
+ Fri, 8 Aug 2014 11:19:03 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-2.mit.edu (dmz-mailsec-scanner-2.mit.edu\r
+ [18.9.25.13])\r
+ (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 0F853431FAE\r
+ for <notmuch@notmuchmail.org>; Fri, 8 Aug 2014 11:19:02 -0700 (PDT)\r
+X-AuditID: 1209190d-f79c06d000002f07-5e-53e51496d054\r
+Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])\r
+ (using TLS with cipher AES256-SHA (256/256 bits))\r
+ (Client did not present a certificate)\r
+ by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id BA.A9.12039.69415E35; Fri, 8 Aug 2014 14:19:02 -0400 (EDT)\r
+Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
+ by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id s78IJ0xw029781; \r
+ Fri, 8 Aug 2014 14:19:01 -0400\r
+Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s78IIwn2007149\r
+ (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
+ Fri, 8 Aug 2014 14:19:00 -0400\r
+Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
+ (envelope-from <amdragon@MIT.EDU>)\r
+ id 1XFokk-0001dF-2r; Fri, 08 Aug 2014 14:18:58 -0400\r
+Date: Fri, 8 Aug 2014 14:18:57 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: Tomi Ollila <tomi.ollila@iki.fi>\r
+Subject: Re: [PATCH v3 00/13] Implement and use database "features"\r
+Message-ID: <20140808181856.GA17169@mit.edu>\r
+References: <1406859003-11561-1-git-send-email-amdragon@mit.edu>\r
+ <m2a97gt15y.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Content-Disposition: inline\r
+In-Reply-To: <m2a97gt15y.fsf@guru.guru-group.fi>\r
+User-Agent: Mutt/1.5.21 (2010-09-15)\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFmpmleLIzCtJLcpLzFFi42IRYrdT150m8jTYYOUkIYvrN2cyW7xZOY/V\r
+ gcnj8NeFLB7PVt1iDmCK4rJJSc3JLEst0rdL4Mr4cXUOU8E1hYrfz9+yNTA+kuxi5OSQEDCR\r
+ 2NC8jQnCFpO4cG89G4gtJDCbSeLzitQuRi4gewOjxMo7jUwQzikmif2rDjFDOEsYJeb/2ADW\r
+ ziKgIvGjZxZYO5uAhsS2/csZQWwRoPiDtvWsIDazgLTEt9/NYPXCAs4Sp442soPYvAI6EjsX\r
+ 9rFCrE6TeLntBjNEXFDi5MwnLBC9WhI3/r0E6uUAm7P8HwdImFPAQGLe8Xlg5aJAq6ac3MY2\r
+ gVFoFpLuWUi6ZyF0L2BkXsUom5JbpZubmJlTnJqsW5ycmJeXWqRrpJebWaKXmlK6iREU1pyS\r
+ vDsY3x1UOsQowMGoxMMr0P04WIg1say4MvcQoyQHk5Io7w7+p8FCfEn5KZUZicUZ8UWlOanF\r
+ hxglOJiVRHifPHoSLMSbklhZlVqUD5OS5mBREud9a20VLCSQnliSmp2aWpBaBJOV4eBQkuC9\r
+ Jgw0VLAoNT21Ii0zpwQhzcTBCTKcB2j4JZAa3uKCxNzizHSI/ClGRSlxXluQhABIIqM0D64X\r
+ lnZeMYoDvSLMux2kigeYsuC6XwENZgIaLKv6GGRwSSJCSqqBUUEvLi/BcuqhnLnST7h6sv63\r
+ LbVLTOBbHHD0idSvXcbS9kctvB9tm2FZa8/qxJEiKLn6c1/UNCOOKfe69W+niydq6N1SWVXf\r
+ Hcu/MlW7fKWU346yMwciK67OK03N27jP9tYWpfynfGv35BXdijrsvP/14xNR2o9Xa27cx3hY\r
+ npn9/e2K9VPjlViKMxINtZiLihMBI6E+yRYDAAA=\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 08 Aug 2014 18:19:11 -0000\r
+\r
+Quoth Tomi Ollila on Aug 08 at 12:55 am:\r
+> On Fri, Aug 01 2014, Austin Clements <amdragon@MIT.EDU> wrote:\r
+> \r
+> > This is v3 of id:1406652492-27803-1-git-send-email-amdragon@mit.edu.\r
+> > This fixes one issue and tidies up another thing in\r
+> > notmuch_database_upgrade I found while working on change tracking\r
+> > support. Most of the patches are logically identical to v2, but the\r
+> > changes ripple through the patch context, so I'm sending a new series.\r
+> >\r
+> > First, this updates notmuch->features before starting the upgrade,\r
+> > rather than after, so that functions called by upgrade will use the\r
+> > new database features instead of the old (this didn't matter in this\r
+> > series because nothing modified the database differently depending on\r
+> > features). Second, this combines multiple _notmuch_message_sync calls\r
+> > into one, which cleans up the code, should further improve upgrade\r
+> > performance, and makes way for additional per-message upgrades.\r
+> \r
+> This series looks good to me. I looked through the diffs a few times and\r
+> notmuch_database_upgrade() in lib/database.cc to see that in full.\r
+> Tests pass (also T530-upgrade now that I downloaded that one test database.)\r
+> \r
+> I googled answers to few questions along the review; one thing still\r
+> interests me -- is there potential to have speed/memory problems\r
+> when doing upgrade transaction in large/very large databases. And\r
+> how long will the (final) commit_transaction() take (i.e how\r
+> many times handle_sigalrm() is called while that is in progress...)\r
+> \r
+> (my guess is that this transaction just builds a new revision and\r
+> if it is never committed the revision is never used -- and data is\r
+> written there in some batches of suitable size -- so memory usage\r
+> and transaction commit time is O(1))\r
+\r
+Your guess is basically right. Xapian periodically flushes stuff to\r
+disk during a transaction basically just like it does when not in a\r
+transaction; AFAIK the only difference is when it flushes out the new\r
+revision number and updated free block metadata. Hence, speed and\r
+memory use aren't affected by the use of a transaction, and\r
+commit_transaction isn't particularly sensitive to how big the\r
+transaction is.\r
+\r
+> Tomi\r
+> \r
+> >\r
+> > The diff from v2 is below (excluding patch 2, which David pushed).\r
+> >\r
+> > diff --git a/lib/database.cc b/lib/database.cc\r
+> > index b323691..d90a924 100644\r
+> > --- a/lib/database.cc\r
+> > +++ b/lib/database.cc\r
+> > @@ -1252,6 +1252,10 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,\r
+> > /* Perform the upgrade in a transaction. */\r
+> > db->begin_transaction (true);\r
+> > \r
+> > + /* Set the target features so we write out changes in the desired\r
+> > + * format. */\r
+> > + notmuch->features = target_features;\r
+> > +\r
+> > /* Perform per-message upgrades. */\r
+> > if (new_features &\r
+> > (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER)) {\r
+> > @@ -1280,7 +1284,6 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,\r
+> > if (filename && *filename != '\0') {\r
+> > _notmuch_message_add_filename (message, filename);\r
+> > _notmuch_message_clear_data (message);\r
+> > - _notmuch_message_sync (message);\r
+> > }\r
+> > talloc_free (filename);\r
+> > }\r
+> > @@ -1289,10 +1292,10 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,\r
+> > * probabilistic and stemmed. Change it to the current\r
+> > * boolean prefix. Add "path:" prefixes while at it.\r
+> > */\r
+> > - if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER) {\r
+> > + if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER)\r
+> > _notmuch_message_upgrade_folder (message);\r
+> > - _notmuch_message_sync (message);\r
+> > - }\r
+> > +\r
+> > + _notmuch_message_sync (message);\r
+> > \r
+> > notmuch_message_destroy (message);\r
+> > \r
+> > @@ -1348,7 +1351,6 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,\r
+> > }\r
+> > }\r
+> > \r
+> > - notmuch->features = target_features;\r
+> > db->set_metadata ("features", _print_features (local, notmuch->features));\r
+> > db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION));\r
+> >\r
projects / notmuch-archives.git / commitdiff