'notmuch new' leaking memory and getting slower over time?

diff --git a/5a/b0e18223949e8e05c7f6116570f96518c6b5da b/5a/b0e18223949e8e05c7f6116570f96518c6b5da
new file mode 100644 (file)
index 0000000..2872fa9
--- /dev/null
+++ b/5a/b0e18223949e8e05c7f6116570f96518c6b5da
@@ -0,0 +1,103 @@
+Return-Path: <pre@ulrik.uio.no>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 3D50F431FD0\r
+       for <notmuch@notmuchmail.org>; Mon, 21 Nov 2011 14:35:40 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.3\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id zE8LyTLv-IVn for <notmuch@notmuchmail.org>;\r
+       Mon, 21 Nov 2011 14:35:39 -0800 (PST)\r
+Received: from mail-out2.uio.no (mail-out2.uio.no [129.240.10.58])\r
+       (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id 630E8431FB6\r
+       for <notmuch@notmuchmail.org>; Mon, 21 Nov 2011 14:35:39 -0800 (PST)\r
+Received: from mail-mx4.uio.no ([129.240.10.45])\r
+       by mail-out2.uio.no with esmtp (Exim 4.75)\r
+       (envelope-from <pre@ulrik.uio.no>) id 1RScSf-00027O-Lx\r
+       for notmuch@notmuchmail.org; Mon, 21 Nov 2011 23:35:37 +0100\r
+Received: from diskless.uio.no ([129.240.6.26])\r
+       by mail-mx4.uio.no with esmtp  (Exim 4.76)\r
+       (envelope-from <pre@ulrik.uio.no>)\r
+       id 1RScSf-0007Sv-8r; Mon, 21 Nov 2011 23:35:37 +0100\r
+Received: from pre by diskless.uio.no with local (Exim 4.72)\r
+       (envelope-from <pre@ulrik.uio.no>)\r
+       id 1RScSd-0004aB-RT; Mon, 21 Nov 2011 23:35:35 +0100\r
+From: Petter Reinholdtsen <pere@hungry.com>\r
+To: notmuch@notmuchmail.org\r
+Subject: 'notmuch new' leaking memory and getting slower over time?\r
+User-Agent: Notmuch/0.10~rc2 (http://notmuchmail.org) Emacs/23.2.1\r
+       (i486-pc-linux-gnu)\r
+Date: Mon, 21 Nov 2011 23:35:34 +0100\r
+Message-ID: <2flfwhht87d.fsf@diskless.uio.no>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Sender: Petter Reinholdtsen <petter.reinholdtsen@usit.uio.no>\r
+X-UiO-Ratelimit-Test: rcpts/h 4 msgs/h 2 sum rcpts/h 7 sum msgs/h 3 total\r
+       rcpts 17725 max rcpts/h 383 ratelimit 0\r
+X-UiO-Spam-info: not spam, SpamAssassin (score=-7.0, required=5.0,\r
+       autolearn=disabled, RP_MATCHES_RCVD=-2.023,\r
+       UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO)\r
+X-UiO-Scanned: C1E0D43C16D4E18FA3AA3F4EF4FCD48C2811D16C\r
+X-UiO-SPAM-Test: remote_host: 129.240.6.26 spam_score: -69 maxlevel 80\r
+       minaction 0 bait 0 mail/h: 3 total 1310 max/h 10 blacklist 0\r
+       greylist 0 ratelimit 0\r
+X-Mailman-Approved-At: Tue, 22 Nov 2011 08:49:56 -0800\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 21 Nov 2011 22:35:40 -0000\r
+\r
+\r
+This weekend up updated my notmuch version to the 0.10 rc2 version\r
+available in Debian/unstable, and made a few observations I would like\r
+to share.  I rebuilt the source on Debian/Squeeze to get it working for\r
+the machine I use to read email, and patched it to get the unsorted dump\r
+before building it.  The reason is that my mail store contain slightly\r
+more than 1.2 million emails at the moment, and just dumping the tags\r
+take hours without the unsorted dump patch.  With the patch it took 40\r
+minutes.  After dumping the tags, I moved away the .notmuch index and\r
+started reindexing using 'notmuch new'.\r
+\r
+The indexing took 36 hours.  At the start it claimed it would take 10\r
+hours, and it continued to underestimate the amount of time left until\r
+the very end.  It claimed to have 1 hour left when I checked before I\r
+went to bed, and claimed to have 15 minutes left when I woke up 6-7\r
+hours later.\r
+\r
+Shortly before the indexing finished, the notmuch process was using 1.2\r
+GiB of resident memory according to top.  Is the process leaking memory?\r
+\r
+Running 'notmuch restore' to get my tags back took 106 minutes, and I\r
+was very surprised that the restore could not load all the tags stored\r
+by 'notmuch dump'.  The restore complained about this line:\r
+\r
+  NO*TELEMAX**NORWAYII  M0018001012307699038 (unread usit year-2002)\r
+\r
+The message in question is a bounce from some X400 mail system, and its\r
+message id look like this:\r
+\r
+  Message-Id: <NO*TELEMAX**NORWAYII    M0018001012307699038>\r
+\r
+I would like 'notmuch new' to use less memory and be better at\r
+estimating the time left, and would also like 'notmuch restore' to\r
+always be able to load the output from 'notmuch dump'.\r
+-- \r
+Happy hacking\r
+Petter Reinholdtsen\r