From 9105b262d0bc210033a47ade541e54411838ece1 Mon Sep 17 00:00:00 2001 From: Mark Walters Date: Mon, 23 Jun 2014 21:12:29 +0100 Subject: [PATCH] [PATCH] dump: make dump take Xapian write lock --- c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e | 126 ++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e diff --git a/c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e b/c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e new file mode 100644 index 000000000..a10fadbc4 --- /dev/null +++ b/c2/fb8ba6dd88bc5ff8c1102e41fcaffa9feae43e @@ -0,0 +1,126 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id CED26431FB6 + for ; Mon, 23 Jun 2014 13:12:46 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0.201 +X-Spam-Level: +X-Spam-Status: No, score=0.201 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + FREEMAIL_ENVFROM_END_DIGIT=1, FREEMAIL_FROM=0.001, + RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id KZhO+mMxiubE for ; + Mon, 23 Jun 2014 13:12:42 -0700 (PDT) +Received: from mail-we0-f176.google.com (mail-we0-f176.google.com + [74.125.82.176]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 79CFC431FAF + for ; Mon, 23 Jun 2014 13:12:42 -0700 (PDT) +Received: by mail-we0-f176.google.com with SMTP id u56so7781609wes.35 + for ; Mon, 23 Jun 2014 13:12:40 -0700 (PDT) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; + h=from:to:cc:subject:date:message-id:in-reply-to:references; + bh=x7pbQ1dkhjnOGE2vFTIL3+Rp69xxQzGhYctHkRhegB4=; + b=dZp6dnlVd0N3vVrRUEfTMk4E1J1dYRB2ZZUL5AaMD3AaA6YP++ax8AVaAtk4gpOaDS + bOIFcPaCIn+EmrBYT+KVkBviBMd72J9+6XST84I31Etpyo5dgybNJvFDx1y6cb6HcnsY + hn5FV/8gQe2fMJoy49AE1rSC2p+kP8NVtLc/aBsuUqBlXeHSP8wdMpzjt7Ksgu1x8DCk + 1h2ts4y9W/94Ty7RvmSmNGPFOOKZBjCs+zT6He8EiKbU6xQWG++j1cUgerN7zAwjpIA3 + WXf0TR/+op1oKIjfsIJvzhcPfV7u1RIaxCwe3WF38y4ibHN6j8BJyshXRRHxfWl+Z+F3 + DZRQ== +X-Received: by 10.194.90.106 with SMTP id bv10mr31064118wjb.20.1403554359229; + Mon, 23 Jun 2014 13:12:39 -0700 (PDT) +Received: from localhost (5751dfa2.skybroadband.com. [87.81.223.162]) + by mx.google.com with ESMTPSA id + fb15sm36378270wid.23.2014.06.23.13.12.38 for + (version=TLSv1.2 cipher=RC4-SHA bits=128/128); + Mon, 23 Jun 2014 13:12:38 -0700 (PDT) +From: Mark Walters +To: notmuch@notmuchmail.org +Subject: [PATCH] dump: make dump take Xapian write lock +Date: Mon, 23 Jun 2014 21:12:29 +0100 +Message-Id: <1403554349-8888-1-git-send-email-markwalters1009@gmail.com> +X-Mailer: git-send-email 1.7.10.4 +In-Reply-To: <87zjhh67e7.fsf@qmul.ac.uk> +References: <87zjhh67e7.fsf@qmul.ac.uk> +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 23 Jun 2014 20:12:47 -0000 + +Dump currently only takes the read lock. Xapian can cope with some +changes while maintaining a read snapshot but with more changes it +fails. Currently notmuch just gives a xapian error. + +To avoid this we take the write lock when dumping. This prevents other +notmuch processes from modifying the xapian database preventing this +error. + +Discussion with Olly on irc indicates that this is currently the best +solution: in xapian trunk there may be better possibilities using +snapshots but they need to make it to a release and propogate out to +users before we can switch approach. + +Finally, this breaks one use case: pipelines of the form + +notmuch dump | ... | notmuch restore + +According to Olly this is already very fragile: it will only work on +small databases. One of the tests relies on this behaviour so fix that +to store the dump rather than use a pipe. +--- +I haven't tested this much: all tests pass (with the one fix) and it is simple. + +Best wishes + +Mark + + + notmuch-dump.c | 2 +- + test/T150-tagging.sh | 4 ++-- + 2 files changed, 3 insertions(+), 3 deletions(-) + +diff --git a/notmuch-dump.c b/notmuch-dump.c +index 887a208..9c6ad7f 100644 +--- a/notmuch-dump.c ++++ b/notmuch-dump.c +@@ -212,7 +212,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[]) + int ret; + + if (notmuch_database_open (notmuch_config_get_database_path (config), +- NOTMUCH_DATABASE_MODE_READ_ONLY, ¬much)) ++ NOTMUCH_DATABASE_MODE_READ_WRITE, ¬much)) + return EXIT_FAILURE; + + char *output_file_name = NULL; +diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh +index dc118f3..45471ac 100755 +--- a/test/T150-tagging.sh ++++ b/test/T150-tagging.sh +@@ -247,8 +247,8 @@ ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \ + notmuch dump --format=batch-tag | sed 's/^.* -- /+common_tag -- /' | \ + sort > EXPECTED + +-notmuch dump --format=batch-tag | sed 's/^.* -- / -- /' | \ +- notmuch restore --format=batch-tag ++notmuch dump --format=batch-tag | sed 's/^.* -- / -- /' > INTERMEDIATE_STEP ++notmuch restore --format=batch-tag < INTERMEDIATE_STEP + + notmuch tag --batch < EXPECTED + +-- +1.7.10.4 + -- 2.26.2