From: Mark Walters Date: Sun, 25 Nov 2012 16:15:50 +0000 (+0000) Subject: Re: [PATCH v2 15/20] insert: fsync new directory after rename X-Git-Url: http://git.tremily.us/gitweb.cgi?a=commitdiff_plain;h=eba505b94455e64eb5144c8d5c5ecd43f6bdee82;p=notmuch-archives.git Re: [PATCH v2 15/20] insert: fsync new directory after rename --- diff --git a/4b/ef1485b1606a852c17caa8ff90bf5937306ab2 b/4b/ef1485b1606a852c17caa8ff90bf5937306ab2 new file mode 100644 index 000000000..b47ddecdb --- /dev/null +++ b/4b/ef1485b1606a852c17caa8ff90bf5937306ab2 @@ -0,0 +1,197 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 99B81431FAF + for ; Sun, 25 Nov 2012 08:15:55 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -1.098 +X-Spam-Level: +X-Spam-Status: No, score=-1.098 tagged_above=-999 required=5 + tests=[DKIM_ADSP_CUSTOM_MED=0.001, FREEMAIL_FROM=0.001, + NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id vSwYyDBotsF6 for ; + Sun, 25 Nov 2012 08:15:53 -0800 (PST) +Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id BCD32431FAE + for ; Sun, 25 Nov 2012 08:15:52 -0800 (PST) +Received: from smtp.qmul.ac.uk ([138.37.6.40]) + by mail2.qmul.ac.uk with esmtp (Exim 4.71) + (envelope-from ) + id 1Tcerz-00065i-T6; Sun, 25 Nov 2012 16:15:48 +0000 +Received: from 93-97-24-31.zone5.bethere.co.uk ([93.97.24.31] helo=localhost) + by smtp.qmul.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) + (envelope-from ) + id 1Tcerz-0001Wa-H7; Sun, 25 Nov 2012 16:15:47 +0000 +From: Mark Walters +To: Peter Wang , notmuch@notmuchmail.org +Subject: Re: [PATCH v2 15/20] insert: fsync new directory after rename +In-Reply-To: <1353806206-29133-16-git-send-email-novalazy@gmail.com> +References: <1353806206-29133-1-git-send-email-novalazy@gmail.com> + <1353806206-29133-16-git-send-email-novalazy@gmail.com> +User-Agent: Notmuch/0.14+81~g9730584 (http://notmuchmail.org) Emacs/23.4.1 + (x86_64-pc-linux-gnu) +Date: Sun, 25 Nov 2012 16:15:50 +0000 +Message-ID: <87624tk6a1.fsf@qmul.ac.uk> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-Sender-Host-Address: 93.97.24.31 +X-QM-SPAM-Info: Sender has good ham record. :) +X-QM-Body-MD5: dd4d33770611628382466e9c89ac22ec (of first 20000 bytes) +X-SpamAssassin-Score: -1.8 +X-SpamAssassin-SpamBar: - +X-SpamAssassin-Report: The QM spam filters have analysed this message to + determine if it is + spam. We require at least 5.0 points to mark a message as spam. + This message scored -1.8 points. + Summary of the scoring: + * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, + * medium trust + * [138.37.6.40 listed in list.dnswl.org] + * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail + provider * (markwalters1009[at]gmail.com) + * 0.5 AWL AWL: From: address is in the auto white-list +X-QM-Scan-Virus: ClamAV says the message is clean +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Sun, 25 Nov 2012 16:15:55 -0000 + +On Sun, 25 Nov 2012, Peter Wang wrote: +> After moving the file from the 'tmp' to the 'new' directory, +> fsync on the 'new' directory for durability. +> --- +> notmuch-insert.c | 39 ++++++++++++++++++++++++++++++++------- +> 1 file changed, 32 insertions(+), 7 deletions(-) +> +> diff --git a/notmuch-insert.c b/notmuch-insert.c +> index f09c579..831b322 100644 +> --- a/notmuch-insert.c +> +++ b/notmuch-insert.c +> @@ -143,10 +143,10 @@ maildir_create_folder (void *ctx, const char *dir) +> /* Open a unique file in the Maildir 'tmp' directory. +> * Returns the file descriptor on success, or -1 on failure. +> * On success, file paths into the 'tmp' and 'new' directories are returned +> - * via tmppath and newpath. */ +> + * via tmppath and newpath, and the path of the 'new' directory in newdir. */ + +I found this comment very difficult to understand since it looks like +the 'new' directory gets returned twice. Perhaps something like + +"On success, file paths for the message in the 'tmp' and 'new' +directories are returned via tmppath and newpath, and the path of the +'new' directory itself in newdir." + + +Best wishes + +Mark + +> static int +> maildir_open_tmp_file (void *ctx, const char *dir, +> - char **tmppath, char **newpath) +> + char **tmppath, char **newpath, char **newdir) +> { +> pid_t pid; +> char hostname[256]; +> @@ -183,8 +183,9 @@ maildir_open_tmp_file (void *ctx, const char *dir, +> return -1; +> } +> +> + *newdir = talloc_asprintf (ctx, "%s/new", dir); +> *newpath = talloc_asprintf (ctx, "%s/new/%s", dir, filename); +> - if (! *newpath) { +> + if (! *newdir || ! *newpath) { +> fprintf (stderr, "Out of memory\n"); +> close (fd); +> unlink (*tmppath); +> @@ -204,14 +205,31 @@ maildir_open_tmp_file (void *ctx, const char *dir, +> * http://wiki.dovecot.org/MailboxFormat/Maildir#Mail_delivery +> */ +> static notmuch_bool_t +> -maildir_move_tmp_to_new (const char *tmppath, const char *newpath) +> +maildir_move_tmp_to_new (const char *tmppath, const char *newpath, +> + const char *newdir) +> { +> + notmuch_bool_t ret; +> + int fd; +> + +> if (rename (tmppath, newpath) != 0) { +> fprintf (stderr, "Error: rename() failed: %s\n", strerror (errno)); +> return FALSE; +> } +> +> - return TRUE; +> + /* Sync the 'new' directory after rename for durability. */ +> + ret = TRUE; +> + fd = open (newdir, O_RDONLY); +> + if (fd == -1) { +> + fprintf (stderr, "Error: open() dir failed: %s\n", strerror (errno)); +> + ret = FALSE; +> + } +> + if (ret && fsync (fd) != 0) { +> + fprintf (stderr, "Error: fsync() dir failed: %s\n", strerror (errno)); +> + ret = FALSE; +> + } +> + if (fd != -1) +> + close (fd); +> + return ret; +> } +> +> /* Copy the contents of fdin into fdout. */ +> @@ -307,6 +325,12 @@ add_file_to_database (notmuch_database_t *notmuch, const char *path, +> +> notmuch_message_thaw (message); +> +> + /* notmuch_message_tags_to_maildir_flags may rename the message file +> + * once more, and does so without fsyncing the directory afterwards. +> + * rename() is atomic so after a crash the file should appear under +> + * the old or new name. notmuch new should be able to rename the file +> + * again if required, so another fsync is not required, I think. +> + */ +> notmuch_message_tags_to_maildir_flags (message); +> +> notmuch_message_destroy (message); +> @@ -321,10 +345,11 @@ insert_message (void *ctx, notmuch_database_t *notmuch, int fdin, +> { +> char *tmppath; +> char *newpath; +> + char *newdir; +> int fdout; +> notmuch_bool_t ret; +> +> - fdout = maildir_open_tmp_file (ctx, dir, &tmppath, &newpath); +> + fdout = maildir_open_tmp_file (ctx, dir, &tmppath, &newpath, &newdir); +> if (fdout < 0) { +> return FALSE; +> } +> @@ -335,7 +360,7 @@ insert_message (void *ctx, notmuch_database_t *notmuch, int fdin, +> } +> close (fdout); +> if (ret) { +> - ret = maildir_move_tmp_to_new (tmppath, newpath); +> + ret = maildir_move_tmp_to_new (tmppath, newpath, newdir); +> } +> if (!ret) { +> unlink (tmppath); +> -- +> 1.7.12.1 +> +> _______________________________________________ +> notmuch mailing list +> notmuch@notmuchmail.org +> http://notmuchmail.org/mailman/listinfo/notmuch