--- /dev/null
+Return-Path: <prvs=310563a60d=amthrax@drake.mit.edu>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 2D0DE431FB6\r
+ for <notmuch@notmuchmail.org>; Wed, 4 May 2011 13:32:22 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id ENqVoKGana1U for <notmuch@notmuchmail.org>;\r
+ Wed, 4 May 2011 13:32:21 -0700 (PDT)\r
+Received: from dmz-mailsec-scanner-3.mit.edu (DMZ-MAILSEC-SCANNER-3.MIT.EDU\r
+ [18.9.25.14])\r
+ by olra.theworths.org (Postfix) with ESMTP id 4A1BD431FB5\r
+ for <notmuch@notmuchmail.org>; Wed, 4 May 2011 13:32:21 -0700 (PDT)\r
+X-AuditID: 1209190e-b7c80ae0000047dd-57-4dc1b7d98d6f\r
+Received: from mailhub-auth-2.mit.edu ( [18.7.62.36])\r
+ by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP\r
+ id 52.33.18397.9D7B1CD4; Wed, 4 May 2011 16:32:25 -0400 (EDT)\r
+Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
+ by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id p44KWII8021336; \r
+ Wed, 4 May 2011 16:32:18 -0400\r
+Received: from drake.mit.edu (26-4-180.dynamic.csail.mit.edu [18.26.4.180])\r
+ (authenticated bits=0)\r
+ (User authenticated as amdragon@ATHENA.MIT.EDU)\r
+ by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id p44KWGmm023656\r
+ (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
+ Wed, 4 May 2011 16:32:18 -0400 (EDT)\r
+Received: from amthrax by drake.mit.edu with local (Exim 4.72)\r
+ (envelope-from <amthrax@drake.mit.edu>)\r
+ id 1QHik4-0000bb-AM; Wed, 04 May 2011 16:32:16 -0400\r
+From: Austin Clements <amdragon@MIT.EDU>\r
+To: notmuch@notmuchmail.org\r
+Subject: [PATCH 01/10 v2] test: Test atomicity of notmuch new.\r
+Date: Wed, 4 May 2011 16:32:03 -0400\r
+Message-Id: <1304541123-2237-1-git-send-email-amdragon@mit.edu>\r
+X-Mailer: git-send-email 1.7.4.1\r
+In-Reply-To: <1298015940-31986-2-git-send-email-amdragon@mit.edu>\r
+References: <1298015940-31986-2-git-send-email-amdragon@mit.edu>\r
+X-Brightmail-Tracker:\r
+ H4sIAAAAAAAAA+NgFnrOIsWRmVeSWpSXmKPExsUixG6nontz+0Ffg+8/DSyu35zJ7MDo8WzV\r
+ LeYAxihum6TEkrLgzPQ8fbsE7ozt+3YxFmy0r1h14BpLA2O3aRcjJ4eEgInE9obFLBC2mMSF\r
+ e+vZuhi5OIQE9jFKPJy/kRXCWc8ose3bJijnGJPEqo3XocrmM0qcv3aICaSfTUBDYtv+5Ywg\r
+ toiAtMTOu7OBOjg4mAVEJN7NBCsRFrCT2Hv4BNg6FgFViSt/2llBbF4Be4kVM5+wQpyhILHg\r
+ 3ls2EJtTwFHidMdusJFCAg4Sr3bcZpvAyL+AkWEVo2xKbpVubmJmTnFqsm5xcmJeXmqRrrFe\r
+ bmaJXmpK6SZGUOBwSvLtYPx6UOkQowAHoxIPL1PyQV8h1sSy4srcQ4ySHExKorylm4FCfEn5\r
+ KZUZicUZ8UWlOanFhxglOJiVRHh39ALleFMSK6tSi/JhUtIcLErivDMl1X2FBNITS1KzU1ML\r
+ UotgsjIcHEoSvBe2ATUKFqWmp1akZeaUIKSZODhBhnNJiRSn5qWkFiWWlmTEg2IpvhgYTSAp\r
+ HqC9RttB9hYXJOYCRSFaTzFacjzYcGo/I8flRWeB5LJVQFKIJS8/L1VKnFcHpEEApCGjNA9u\r
+ HSx9vGIUB/pemPcnyFU8wNQDN/UV0EImoIVezQdAFpYkIqSkGhjb87r83sn3b2QtW3kiyXh/\r
+ 84zZykeyHx+Sql72hW3rlTMTyi9xlkcuOh1//QjbEcb8rm6H00smJTzOfr3kPfvBED2L6Gz5\r
+ lj2XBYsORHWriufyezqL7w0tDnv31m6Td+hs3drjipK630JnuvB4Hj1bOyvh2qJVX7rDlY58\r
+ nHv69/X0adnP+o4osRRnJBpqMRcVJwIAzbTj4/oCAAA=\r
+Cc: amdragon@mit.edu\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 04 May 2011 20:32:22 -0000\r
+\r
+This tests notmuch new's ability to recover from arbitrary stopping\r
+failures. It interrupts notmuch new after every database commit and,\r
+on every resulting database snapshot, re-runs notmuch new to\r
+completion and checks that the final database state is invariant.\r
+---\r
+This addresses a timing bug in the atomicity test found by jrollins\r
+and also adds additional comments.\r
+\r
+ test/atomicity | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++\r
+ test/atomicity.gdb | 50 +++++++++++++++++++++++++++\r
+ test/basic | 4 ++-\r
+ test/notmuch-test | 1 +\r
+ 4 files changed, 150 insertions(+), 1 deletions(-)\r
+ create mode 100755 test/atomicity\r
+ create mode 100644 test/atomicity.gdb\r
+\r
+diff --git a/test/atomicity b/test/atomicity\r
+new file mode 100755\r
+index 0000000..b7ae505\r
+--- /dev/null\r
++++ b/test/atomicity\r
+@@ -0,0 +1,96 @@\r
++#!/bin/bash\r
++test_description='atomicity'\r
++. ./test-lib.sh\r
++\r
++# This script tests the effects of killing and restarting "notmuch\r
++# new" at arbitrary points. If notmuch new is properly atomic, the\r
++# final database contents should be the same regardless of when (or\r
++# if) it is killed and restarted.\r
++\r
++# Create a maildir structure to also stress flag synchronization\r
++mkdir $MAIL_DIR/cur\r
++mkdir $MAIL_DIR/new\r
++mkdir $MAIL_DIR/tmp\r
++mkdir $MAIL_DIR/.remove-dir\r
++\r
++# Prepare the initial database\r
++generate_message [subject]='Duplicate' [filename]='duplicate:2,' [dir]=cur\r
++generate_message [subject]='Remove' [filename]='remove:2,' [dir]=cur\r
++generate_message [subject]='"Remove duplicate"' [filename]='remove-duplicate:2,' [dir]=cur\r
++cp $MAIL_DIR/cur/remove-duplicate:2, $MAIL_DIR/cur/remove-duplicate-copy:2,\r
++generate_message [subject]='Rename' [filename]='rename:2,' [dir]=cur\r
++generate_message [subject]='"Rename duplicate"' [filename]='rename-duplicate:2,' [dir]=cur\r
++generate_message [subject]='"Move 1"' [filename]='move1:2,' [dir]=cur\r
++generate_message [subject]='"Move 2"' [filename]='move2:2,' [dir]=new\r
++generate_message [subject]='Flag' [filename]='flag:2,' [dir]=cur\r
++generate_message [subject]='"Flag duplicate"' [filename]='flag-duplicate:2,' [dir]=cur\r
++cp $MAIL_DIR/cur/flag-duplicate:2, $MAIL_DIR/cur/flag-duplicate-copy:2,F\r
++generate_message [subject]='"Remove directory"' [filename]='remove-directory:2,' [dir]=.remove-dir\r
++generate_message [subject]='"Remove directory duplicate"' [filename]='remove-directory-duplicate:2,' [dir]=.remove-dir\r
++cp $MAIL_DIR/.remove-dir/remove-directory-duplicate:2, $MAIL_DIR/cur/\r
++notmuch new > /dev/null\r
++\r
++# Make all maildir changes, but *don't* update the database\r
++generate_message [subject]='Added' [filename]='added:2,' [dir]=cur\r
++cp $MAIL_DIR/cur/duplicate:2, $MAIL_DIR/cur/duplicate-copy:2,\r
++generate_message [subject]='"Add duplicate"' [filename]='add-duplicate:2,' [dir]=cur\r
++generate_message [subject]='"Add duplicate copy"' [filename]='add-duplicate-copy:2,' [dir]=cur\r
++rm $MAIL_DIR/cur/remove:2,\r
++rm $MAIL_DIR/cur/remove-duplicate-copy:2,\r
++mv $MAIL_DIR/cur/rename:2, $MAIL_DIR/cur/renamed:2,\r
++mv $MAIL_DIR/cur/rename-duplicate:2, $MAIL_DIR/cur/renamed-duplicate:2,\r
++mv $MAIL_DIR/cur/move1:2, $MAIL_DIR/new/move1:2,\r
++mv $MAIL_DIR/new/move2:2, $MAIL_DIR/cur/move2:2,\r
++mv $MAIL_DIR/cur/flag:2, $MAIL_DIR/cur/flag:2,F\r
++rm $MAIL_DIR/cur/flag-duplicate-copy:2,F\r
++rm $MAIL_DIR/.remove-dir/remove-directory:2,\r
++rm $MAIL_DIR/.remove-dir/remove-directory-duplicate:2,\r
++rmdir $MAIL_DIR/.remove-dir\r
++increment_mtime $MAIL_DIR/cur\r
++increment_mtime $MAIL_DIR/new\r
++increment_mtime $MAIL_DIR\r
++\r
++# Prepare a snapshot of the updated maildir. The gdb script will\r
++# update the database in this snapshot as it goes.\r
++cp -ra $MAIL_DIR $MAIL_DIR.snap\r
++cp ${NOTMUCH_CONFIG} ${NOTMUCH_CONFIG}.snap\r
++NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap notmuch config set database.path $MAIL_DIR.snap\r
++\r
++\r
++test_begin_subtest '"notmuch new" is idempotent under arbitrary aborts'\r
++\r
++# Execute notmuch new and, at every call to rename, snapshot the\r
++# database, run notmuch new again on the snapshot, and capture the\r
++# results of search.\r
++export MAIL_DIR\r
++gdb -batch -x ../atomicity.gdb notmuch > /dev/null 2>&1\r
++\r
++# Get the final, golden output\r
++notmuch search '*' > expected\r
++\r
++# Check output against golden output\r
++outcount=$(cat outcount)\r
++echo -n > searchall\r
++echo -n > expectall\r
++for ((i = 0; i < $outcount; i++)); do\r
++ if ! cmp -s search.$i expected; then\r
++ # Find the range of interruptions that match this output\r
++ for ((end = $i + 1 ; end < $outcount; end++)); do\r
++ if ! cmp -s search.$i search.$end; then\r
++ break\r
++ fi\r
++ done\r
++ echo "When interrupted after $test/backtrace.$(expr $i - 1) (abort points $i-$(expr $end - 1))" >> searchall\r
++ cat search.$i >> searchall\r
++ cat expected >> expectall\r
++ echo >> searchall\r
++ echo >> expectall\r
++\r
++ i=$(expr $end - 1)\r
++ fi\r
++done\r
++test_expect_equal_failure "$(cat searchall)" "$(cat expectall)"\r
++\r
++test_expect_success "detected $outcount>10 abort points" "test $outcount -gt 10"\r
++\r
++test_done\r
+diff --git a/test/atomicity.gdb b/test/atomicity.gdb\r
+new file mode 100644\r
+index 0000000..fd67525\r
+--- /dev/null\r
++++ b/test/atomicity.gdb\r
+@@ -0,0 +1,50 @@\r
++# This gdb script runs notmuch new and simulates killing and\r
++# restarting notmuch new after every Xapian commit. To simulate this\r
++# more efficiently, this script runs notmuch new and, immediately\r
++# after every Xapian commit, it *pauses* the running notmuch new,\r
++# copies the entire database and maildir to a snapshot directory, and\r
++# executes a full notmuch new on that snapshot, comparing the final\r
++# results with the expected output. It can then resume the paused\r
++# notmuch new, which is still running on the original maildir, and\r
++# repeat this process.\r
++\r
++set args new\r
++\r
++# Make Xapian commit after every operation instead of batching\r
++set environment XAPIAN_FLUSH_THRESHOLD = 1\r
++\r
++# gdb can't keep track of a simple integer. This is me weeping.\r
++shell echo 0 > outcount\r
++\r
++shell touch inodes\r
++\r
++break rename\r
++commands\r
++# As an optimization, only consider snapshots after a Xapian commit.\r
++# Xapian overwrites record.base? as the last step in the commit.\r
++shell echo > gdbcmd\r
++shell stat -c %i $MAIL_DIR/.notmuch/xapian/record.base* > inodes.new\r
++shell if cmp inodes inodes.new; then echo cont > gdbcmd; fi\r
++shell mv inodes.new inodes\r
++source gdbcmd\r
++\r
++# Save a backtrace in case the test does fail\r
++set logging file backtrace\r
++set logging on\r
++backtrace\r
++set logging off\r
++shell mv backtrace backtrace.`cat outcount`\r
++\r
++# Snapshot the database\r
++shell rm -r $MAIL_DIR.snap/.notmuch\r
++shell cp -r $MAIL_DIR/.notmuch $MAIL_DIR.snap/.notmuch\r
++# Restore the mtime of $MAIL_DIR.snap, which we just changed\r
++shell touch -r $MAIL_DIR $MAIL_DIR.snap\r
++# Run notmuch new to completion on the snapshot\r
++shell NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap XAPIAN_FLUSH_THRESHOLD=1000 notmuch new > /dev/null\r
++shell NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap notmuch search '*' > search.`cat outcount` 2>&1\r
++shell echo $(expr $(cat outcount) + 1) > outcount\r
++cont\r
++end\r
++\r
++run\r
+diff --git a/test/basic b/test/basic\r
+index 3b43ad9..f9d5fc3 100755\r
+--- a/test/basic\r
++++ b/test/basic\r
+@@ -57,7 +57,9 @@ available=$(ls -1 ../ | \\r
+ sed -r -e "/^(aggregate-results.sh|Makefile|Makefile.local|notmuch-test)/d" \\r
+ -e "/^(README|test-lib.sh|test-results|tmp.*|valgrind|corpus*)/d" \\r
+ -e "/^(emacs.expected-output|smtp-dummy|smtp-dummy.c|test-verbose)/d" \\r
+- -e "/^(test.expected-output|.*~)/d" | sort)\r
++ -e "/^(test.expected-output|.*~)/d" \\r
++ -e "/^(atomicity.gdb)/d" \\r
++ | sort)\r
+ test_expect_equal "$tests_in_suite" "$available"\r
+ \r
+ EXPECTED=../test.expected-output\r
+diff --git a/test/notmuch-test b/test/notmuch-test\r
+index 8dd9c1e..5b22fa8 100755\r
+--- a/test/notmuch-test\r
++++ b/test/notmuch-test\r
+@@ -38,6 +38,7 @@ TESTS="\r
+ emacs\r
+ emacs-large-search-buffer\r
+ maildir-sync\r
++ atomicity\r
+ "\r
+ \r
+ # Clean up any results from a previous run\r
+-- \r
+1.7.4.1\r
+\r
projects / notmuch-archives.git / commitdiff