[PATCH] dump: make dump take Xapian write lock

diff --git a/c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e b/c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e
new file mode 100644 (file)
index 0000000..a10fadb
--- /dev/null
+++ b/c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e
@@ -0,0 +1,126 @@
+Return-Path: <markwalters1009@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id CED26431FB6\r
+       for <notmuch@notmuchmail.org>; Mon, 23 Jun 2014 13:12:46 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0.201\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0.201 tagged_above=-999 required=5\r
+       tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,\r
+       FREEMAIL_ENVFROM_END_DIGIT=1, FREEMAIL_FROM=0.001,\r
+       RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id KZhO+mMxiubE for <notmuch@notmuchmail.org>;\r
+       Mon, 23 Jun 2014 13:12:42 -0700 (PDT)\r
+Received: from mail-we0-f176.google.com (mail-we0-f176.google.com\r
+       [74.125.82.176]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 79CFC431FAF\r
+       for <notmuch@notmuchmail.org>; Mon, 23 Jun 2014 13:12:42 -0700 (PDT)\r
+Received: by mail-we0-f176.google.com with SMTP id u56so7781609wes.35\r
+       for <notmuch@notmuchmail.org>; Mon, 23 Jun 2014 13:12:40 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;\r
+       h=from:to:cc:subject:date:message-id:in-reply-to:references;\r
+       bh=x7pbQ1dkhjnOGE2vFTIL3+Rp69xxQzGhYctHkRhegB4=;\r
+       b=dZp6dnlVd0N3vVrRUEfTMk4E1J1dYRB2ZZUL5AaMD3AaA6YP++ax8AVaAtk4gpOaDS\r
+       bOIFcPaCIn+EmrBYT+KVkBviBMd72J9+6XST84I31Etpyo5dgybNJvFDx1y6cb6HcnsY\r
+       hn5FV/8gQe2fMJoy49AE1rSC2p+kP8NVtLc/aBsuUqBlXeHSP8wdMpzjt7Ksgu1x8DCk\r
+       1h2ts4y9W/94Ty7RvmSmNGPFOOKZBjCs+zT6He8EiKbU6xQWG++j1cUgerN7zAwjpIA3\r
+       WXf0TR/+op1oKIjfsIJvzhcPfV7u1RIaxCwe3WF38y4ibHN6j8BJyshXRRHxfWl+Z+F3\r
+       DZRQ==\r
+X-Received: by 10.194.90.106 with SMTP id bv10mr31064118wjb.20.1403554359229; \r
+       Mon, 23 Jun 2014 13:12:39 -0700 (PDT)\r
+Received: from localhost (5751dfa2.skybroadband.com. [87.81.223.162])\r
+       by mx.google.com with ESMTPSA id\r
+       fb15sm36378270wid.23.2014.06.23.13.12.38 for <multiple recipients>\r
+       (version=TLSv1.2 cipher=RC4-SHA bits=128/128);\r
+       Mon, 23 Jun 2014 13:12:38 -0700 (PDT)\r
+From: Mark Walters <markwalters1009@gmail.com>\r
+To: notmuch@notmuchmail.org\r
+Subject: [PATCH] dump: make dump take Xapian write lock\r
+Date: Mon, 23 Jun 2014 21:12:29 +0100\r
+Message-Id: <1403554349-8888-1-git-send-email-markwalters1009@gmail.com>\r
+X-Mailer: git-send-email 1.7.10.4\r
+In-Reply-To: <87zjhh67e7.fsf@qmul.ac.uk>\r
+References: <87zjhh67e7.fsf@qmul.ac.uk>\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 23 Jun 2014 20:12:47 -0000\r
+\r
+Dump currently only takes the read lock. Xapian can cope with some\r
+changes while maintaining a read snapshot but with more changes it\r
+fails. Currently notmuch just gives a xapian error.\r
+\r
+To avoid this we take the write lock when dumping. This prevents other\r
+notmuch processes from modifying the xapian database preventing this\r
+error.\r
+\r
+Discussion with Olly on irc indicates that this is currently the best\r
+solution: in xapian trunk there may be better possibilities using\r
+snapshots but they need to make it to a release and propogate out to\r
+users before we can switch approach.\r
+\r
+Finally, this breaks one use case: pipelines of the form\r
+\r
+notmuch dump | ... | notmuch restore\r
+\r
+According to Olly this is already very fragile: it will only work on\r
+small databases. One of the tests relies on this behaviour so fix that\r
+to store the dump rather than use a pipe.\r
+---\r
+I haven't tested this much: all tests pass (with the one fix) and it is simple.\r
+\r
+Best wishes\r
+\r
+Mark\r
+\r
+\r
+ notmuch-dump.c       |    2 +-\r
+ test/T150-tagging.sh |    4 ++--\r
+ 2 files changed, 3 insertions(+), 3 deletions(-)\r
+\r
+diff --git a/notmuch-dump.c b/notmuch-dump.c\r
+index 887a208..9c6ad7f 100644\r
+--- a/notmuch-dump.c\r
++++ b/notmuch-dump.c\r
+@@ -212,7 +212,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])\r
+     int ret;\r
+ \r
+     if (notmuch_database_open (notmuch_config_get_database_path (config),\r
+-                             NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))\r
++                             NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))\r
+       return EXIT_FAILURE;\r
+ \r
+     char *output_file_name = NULL;\r
+diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh\r
+index dc118f3..45471ac 100755\r
+--- a/test/T150-tagging.sh\r
++++ b/test/T150-tagging.sh\r
+@@ -247,8 +247,8 @@ ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \\r
+ notmuch dump --format=batch-tag | sed 's/^.* -- /+common_tag -- /' | \\r
+     sort > EXPECTED\r
+ \r
+-notmuch dump --format=batch-tag | sed 's/^.* -- /  -- /' | \\r
+-    notmuch restore --format=batch-tag\r
++notmuch dump --format=batch-tag | sed 's/^.* -- /  -- /' > INTERMEDIATE_STEP\r
++notmuch restore --format=batch-tag < INTERMEDIATE_STEP\r
+ \r
+ notmuch tag --batch < EXPECTED\r
+ \r
+-- \r
+1.7.10.4\r
+\r