--- /dev/null
+Return-Path: <jani@nikula.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 4A988431FB6\r
+ for <notmuch@notmuchmail.org>; Sun, 1 Sep 2013 08:43:11 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id RNmHTXttIMd2 for <notmuch@notmuchmail.org>;\r
+ Sun, 1 Sep 2013 08:43:05 -0700 (PDT)\r
+Received: from mail-ee0-f42.google.com (mail-ee0-f42.google.com\r
+ [74.125.83.42]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client\r
+ certificate requested) by olra.theworths.org (Postfix) with ESMTPS id\r
+ F3175431FAE for <notmuch@notmuchmail.org>; Sun, 1 Sep 2013 08:43:04 -0700\r
+ (PDT)\r
+Received: by mail-ee0-f42.google.com with SMTP id b45so1889931eek.1\r
+ for <notmuch@notmuchmail.org>; Sun, 01 Sep 2013 08:43:03 -0700 (PDT)\r
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+ d=1e100.net; s=20130820;\r
+ h=x-gm-message-state:from:to:subject:in-reply-to:references\r
+ :user-agent:date:message-id:mime-version:content-type;\r
+ bh=vibekIws0liU6uhc67V/e7GVfvokv/mcXDZb3z4g5TA=;\r
+ b=gsHmaDM8xfohEkLA6r8m5wmGnKzI/FirKcEzzHuNgMhYx7kEcYoMOvbnv+TuBSxdOn\r
+ hkbBA0p4N8hUb07jAeFT8VO/D0Oo31KzYC/rIcV0DJkNInihNLDVKlJ2WhXs5jOBcHLU\r
+ 8wCYx+sz9HvLMOEfFylVALbXmzweR1s5dTua+sTJbuzs4MwWagacTV7qbRIseNDQyLW5\r
+ x/S12AesDGBB/9Vp9a9t2Z1xJkujwWS9hhKl77cxrZ6oYmOe9OAlLuF9RnVOPRGdG/ny\r
+ yENW6TSKMkQUlyKha5qtqkNWloglbAslectkhC6odwuHOt9HQXei94t/jE9fELw6YRM0\r
+ dYAQ==\r
+X-Gm-Message-State:\r
+ ALoCoQm5bwTgYD2yYOMZ0w/+Wj8WhK5L+3oaksYgcB5XKs52GSEXQfMc23plfVbqrgy2cfX6S77n\r
+X-Received: by 10.14.183.2 with SMTP id p2mr4829714eem.44.1378050183682;\r
+ Sun, 01 Sep 2013 08:43:03 -0700 (PDT)\r
+Received: from localhost (dsl-hkibrasgw2-58c36f-91.dhcp.inet.fi.\r
+ [88.195.111.91])\r
+ by mx.google.com with ESMTPSA id z12sm14238866eev.6.1969.12.31.16.00.00\r
+ (version=TLSv1.2 cipher=RC4-SHA bits=128/128);\r
+ Sun, 01 Sep 2013 08:43:03 -0700 (PDT)\r
+From: Jani Nikula <jani@nikula.org>\r
+To: Ben Gamari <bgamari.foss@gmail.com>, notmuch@notmuchmail.org\r
+Subject: Re: [PATCH 1/3] database: Add notmuch_database_compact_close\r
+In-Reply-To: <1377312923-32274-2-git-send-email-bgamari.foss@gmail.com>\r
+References: <1377312923-32274-1-git-send-email-bgamari.foss@gmail.com>\r
+ <1377312923-32274-2-git-send-email-bgamari.foss@gmail.com>\r
+User-Agent: Notmuch/0.16 (http://notmuchmail.org) Emacs/24.3.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Sun, 01 Sep 2013 18:43:03 +0300\r
+Message-ID: <87ppsssg3c.fsf@nikula.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 01 Sep 2013 15:43:11 -0000\r
+\r
+On Sat, 24 Aug 2013, Ben Gamari <bgamari.foss@gmail.com> wrote:\r
+> This function uses Xapian's Compactor machinery to compact the notmuch\r
+> database. The compacted database is built in a temporary directory and\r
+> later moved into place while the original uncompacted database is\r
+> preserved.\r
+>\r
+> Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>\r
+> ---\r
+> configure | 27 +++++++++++++++--\r
+> lib/database.cc | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\r
+> lib/notmuch.h | 15 ++++++++++\r
+> 3 files changed, 130 insertions(+), 3 deletions(-)\r
+>\r
+> diff --git a/configure b/configure\r
+> index 6166917..ee9e887 100755\r
+> --- a/configure\r
+> +++ b/configure\r
+> @@ -277,7 +277,8 @@ printf "Checking for Xapian development files... "\r
+> have_xapian=0\r
+> for xapian_config in ${XAPIAN_CONFIG}; do\r
+> if ${xapian_config} --version > /dev/null 2>&1; then\r
+> - printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //')\r
+> + xapian_version=$(${xapian_config} --version | sed -e 's/.* //')\r
+> + printf "Yes (%s).\n" ${xapian_version}\r
+> have_xapian=1\r
+> xapian_cxxflags=$(${xapian_config} --cxxflags)\r
+> xapian_ldflags=$(${xapian_config} --libs)\r
+> @@ -289,6 +290,21 @@ if [ ${have_xapian} = "0" ]; then\r
+> errors=$((errors + 1))\r
+> fi\r
+> \r
+> +# Compaction is only supported on Xapian > 1.2.6\r
+> +have_xapian_compact=0\r
+> +if [ ${have_xapian} = "1" ]; then\r
+> + printf "Checking for Xapian compact support... "\r
+> + case "${xapian_version}" in\r
+> + 0.*|1.[01].*|1.2.[0-5])\r
+> + printf "No (only available with Xapian > 1.2.6).\n" ;;\r
+> + [1-9]*.[0-9]*.[0-9]*)\r
+> + have_xapian_compact=1\r
+> + printf "Yes.\n" ;;\r
+> + *)\r
+> + printf "Unknown version.\n" ;;\r
+> + esac\r
+> +fi\r
+> +\r
+> printf "Checking for GMime development files... "\r
+> have_gmime=0\r
+> IFS=';'\r
+> @@ -729,6 +745,9 @@ HAVE_STRCASESTR = ${have_strcasestr}\r
+> # build its own version)\r
+> HAVE_STRSEP = ${have_strsep}\r
+> \r
+> +# Whether the Xapian version in use supports compaction\r
+> +HAVE_XAPIAN_COMPACT = ${have_xapian_compact}\r
+> +\r
+> # Whether the getpwuid_r function is standards-compliant\r
+> # (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS\r
+> # to enable the standards-compliant version -- needed for Solaris)\r
+> @@ -787,13 +806,15 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\\r
+> -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\\r
+> -DHAVE_STRSEP=\$(HAVE_STRSEP) \\\r
+> -DSTD_GETPWUID=\$(STD_GETPWUID) \\\r
+> - -DSTD_ASCTIME=\$(STD_ASCTIME)\r
+> + -DSTD_ASCTIME=\$(STD_ASCTIME) \\\r
+> + -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)\r
+> CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\\r
+> \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\\r
+> \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\\r
+> -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\\r
+> -DHAVE_STRSEP=\$(HAVE_STRSEP) \\\r
+> -DSTD_GETPWUID=\$(STD_GETPWUID) \\\r
+> - -DSTD_ASCTIME=\$(STD_ASCTIME)\r
+> + -DSTD_ASCTIME=\$(STD_ASCTIME) \\\r
+> + -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)\r
+> CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)\r
+> EOF\r
+> diff --git a/lib/database.cc b/lib/database.cc\r
+> index 5cc0765..b71751d 100644\r
+> --- a/lib/database.cc\r
+> +++ b/lib/database.cc\r
+> @@ -268,6 +268,8 @@ notmuch_status_to_string (notmuch_status_t status)\r
+> return "Unbalanced number of calls to notmuch_message_freeze/thaw";\r
+> case NOTMUCH_STATUS_UNBALANCED_ATOMIC:\r
+> return "Unbalanced number of calls to notmuch_database_begin_atomic/end_atomic";\r
+> + case NOTMUCH_STATUS_UNSUPPORTED_OPERATION:\r
+> + return "Unsupported operation";\r
+> default:\r
+> case NOTMUCH_STATUS_LAST_STATUS:\r
+> return "Unknown error status value";\r
+> @@ -800,6 +802,95 @@ notmuch_database_close (notmuch_database_t *notmuch)\r
+> notmuch->date_range_processor = NULL;\r
+> }\r
+> \r
+> +class NotmuchCompactor : public Xapian::Compactor\r
+> +{\r
+> +public:\r
+> + virtual void\r
+> + set_status (const std::string &table, const std::string &status)\r
+> + {\r
+> + if (status.length() == 0)\r
+> + printf ("compacting table %s:\n", table.c_str());\r
+> + else\r
+> + printf (" %s\n", status.c_str());\r
+> + }\r
+\r
+We're trying to reduce the amount of prints directly from libnotmuch,\r
+not increase. This applies here as well as below.\r
+\r
+> +};\r
+> +\r
+> +#if HAVE_XAPIAN_COMPACT\r
+> +notmuch_status_t\r
+> +notmuch_database_compact_close (notmuch_database_t *notmuch)\r
+> +{\r
+> + void *local = talloc_new (NULL);\r
+> + NotmuchCompactor compactor;\r
+> + char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;\r
+> + notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;\r
+> +\r
+> + if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, ".notmuch"))) {\r
+> + ret = NOTMUCH_STATUS_OUT_OF_MEMORY;\r
+> + goto DONE;\r
+> + }\r
+> +\r
+> + if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) {\r
+> + ret = NOTMUCH_STATUS_OUT_OF_MEMORY;\r
+> + goto DONE;\r
+> + }\r
+> +\r
+> + if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", xapian_path))) {\r
+> + ret = NOTMUCH_STATUS_OUT_OF_MEMORY;\r
+> + goto DONE;\r
+> + }\r
+> +\r
+> + if (! (old_xapian_path = talloc_asprintf (local, "%s.old", xapian_path))) {\r
+> + ret = NOTMUCH_STATUS_OUT_OF_MEMORY;\r
+> + goto DONE;\r
+> + }\r
+> +\r
+> + try {\r
+> + compactor.set_renumber(false);\r
+> + compactor.add_source(xapian_path);\r
+> + compactor.set_destdir(compact_xapian_path);\r
+> + compactor.compact();\r
+> +\r
+> + if (rename(xapian_path, old_xapian_path)) {\r
+> + fprintf (stderr, "Error moving old database out of the way\n");\r
+> + ret = NOTMUCH_STATUS_FILE_ERROR;\r
+> + goto DONE;\r
+> + }\r
+\r
+This fails if old_xapian_path exists.\r
+\r
+> +\r
+> + if (rename(compact_xapian_path, xapian_path)) {\r
+> + fprintf (stderr, "Error moving compacted database\n");\r
+> + ret = NOTMUCH_STATUS_FILE_ERROR;\r
+> + goto DONE;\r
+> + }\r
+> + } catch (Xapian::InvalidArgumentError e) {\r
+> + fprintf (stderr, "Error while compacting: %s", e.get_msg().c_str());\r
+> + ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;\r
+> + goto DONE;\r
+> + }\r
+> +\r
+> + fprintf (stderr, "\n");\r
+> + fprintf (stderr, "\n");\r
+> + fprintf (stderr, "Old database has been moved to %s", old_xapian_path);\r
+> + fprintf (stderr, "\n");\r
+> + fprintf (stderr, "To delete run,\n");\r
+> + fprintf (stderr, "\n");\r
+> + fprintf (stderr, " rm -R %s\n", old_xapian_path);\r
+> + fprintf (stderr, "\n");\r
+> +\r
+> + notmuch_database_close(notmuch);\r
+> +\r
+> +DONE:\r
+> + talloc_free(local);\r
+\r
+The database does not get closed on errors. If that's intentional, it\r
+should be documented.\r
+\r
+> + return ret;\r
+> +}\r
+> +#else\r
+> +notmuch_status_t\r
+> +notmuch_database_compact_close (unused (notmuch_database_t *notmuch))\r
+> +{\r
+> + fprintf (stderr, "notmuch was compiled against a xapian version lacking compaction support.\n");\r
+> + return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;\r
+> +}\r
+> +#endif\r
+> +\r
+> void\r
+> notmuch_database_destroy (notmuch_database_t *notmuch)\r
+> {\r
+> diff --git a/lib/notmuch.h b/lib/notmuch.h\r
+> index 998a4ae..e9abd90 100644\r
+> --- a/lib/notmuch.h\r
+> +++ b/lib/notmuch.h\r
+> @@ -101,6 +101,7 @@ typedef enum _notmuch_status {\r
+> NOTMUCH_STATUS_TAG_TOO_LONG,\r
+> NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW,\r
+> NOTMUCH_STATUS_UNBALANCED_ATOMIC,\r
+> + NOTMUCH_STATUS_UNSUPPORTED_OPERATION,\r
+> \r
+> NOTMUCH_STATUS_LAST_STATUS\r
+> } notmuch_status_t;\r
+> @@ -215,6 +216,20 @@ notmuch_database_open (const char *path,\r
+> void\r
+> notmuch_database_close (notmuch_database_t *database);\r
+> \r
+> +/* Close the given notmuch database and then compact it.\r
+\r
+The implementation first compacts then closes.\r
+\r
+> + * After notmuch_database_close_compact has been called, calls to\r
+> + * other functions on objects derived from this database may either\r
+> + * behave as if the database had not been closed (e.g., if the\r
+> + * required data has been cached) or may fail with a\r
+> + * NOTMUCH_STATUS_XAPIAN_EXCEPTION.\r
+> + *\r
+> + * notmuch_database_close_compact can be called multiple times. Later\r
+> + * calls have no effect.\r
+\r
+This is not true. The Xapian compactor does not require the database to\r
+be open. It will happily open the database read-only and compact the\r
+database again if database has been closed.\r
+\r
+> + */\r
+> +notmuch_status_t\r
+> +notmuch_database_compact_close (notmuch_database_t *notmuch);\r
+\r
+I'm afraid we really need to re-think the API.\r
+\r
+I see that your CLI 'notmuch compact' command opens the database\r
+read-write, I assume to ensure there are no other writers, so that stuff\r
+doesn't get lost. However if you pass a read-write database that has\r
+been modified, I believe the changes will get lost (as Xapian opens the\r
+database read-only). We shouldn't let the API users shoot themselves in\r
+the foot so easily.\r
+\r
+I think I'd go for something like:\r
+\r
+notmuch_status_t\r
+notmuch_database_compact (const char *path);\r
+\r
+or\r
+\r
+notmuch_status_t\r
+notmuch_database_compact (const char *path, const char *backup);\r
+\r
+which would internally open the database as read-write to ensure no\r
+modifications, compact, and close. If backup != NULL, it would save the\r
+old database there (same mounted file system as the database is a fine\r
+limitation), otherwise remove.\r
+\r
+Even then I'm not completely sure what Xapian WritableDatabase will do\r
+on close when the database has been switched underneath it. But moving\r
+the database after it has been closed has a race condition too.\r
+\r
+\r
+BR,\r
+Jani.\r
+\r
+\r
+\r
+> +\r
+> /* Destroy the notmuch database, closing it if necessary and freeing\r
+> * all associated resources. */\r
+> void\r
+> -- \r
+> 1.8.1.2\r
+>\r
+> _______________________________________________\r
+> notmuch mailing list\r
+> notmuch@notmuchmail.org\r
+> http://notmuchmail.org/mailman/listinfo/notmuch\r
+\r
+-- \r
+Jani\r
projects / notmuch-archives.git / commitdiff