Return-Path: X-Original-To: notmuch@notmuchmail.org Delivered-To: notmuch@notmuchmail.org Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 3D50F431FD0 for ; Mon, 21 Nov 2011 14:35:40 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -2.3 X-Spam-Level: X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zE8LyTLv-IVn for ; Mon, 21 Nov 2011 14:35:39 -0800 (PST) Received: from mail-out2.uio.no (mail-out2.uio.no [129.240.10.58]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 630E8431FB6 for ; Mon, 21 Nov 2011 14:35:39 -0800 (PST) Received: from mail-mx4.uio.no ([129.240.10.45]) by mail-out2.uio.no with esmtp (Exim 4.75) (envelope-from ) id 1RScSf-00027O-Lx for notmuch@notmuchmail.org; Mon, 21 Nov 2011 23:35:37 +0100 Received: from diskless.uio.no ([129.240.6.26]) by mail-mx4.uio.no with esmtp (Exim 4.76) (envelope-from ) id 1RScSf-0007Sv-8r; Mon, 21 Nov 2011 23:35:37 +0100 Received: from pre by diskless.uio.no with local (Exim 4.72) (envelope-from ) id 1RScSd-0004aB-RT; Mon, 21 Nov 2011 23:35:35 +0100 From: Petter Reinholdtsen To: notmuch@notmuchmail.org Subject: 'notmuch new' leaking memory and getting slower over time? User-Agent: Notmuch/0.10~rc2 (http://notmuchmail.org) Emacs/23.2.1 (i486-pc-linux-gnu) Date: Mon, 21 Nov 2011 23:35:34 +0100 Message-ID: <2flfwhht87d.fsf@diskless.uio.no> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: Petter Reinholdtsen X-UiO-Ratelimit-Test: rcpts/h 4 msgs/h 2 sum rcpts/h 7 sum msgs/h 3 total rcpts 17725 max rcpts/h 383 ratelimit 0 X-UiO-Spam-info: not spam, SpamAssassin (score=-7.0, required=5.0, autolearn=disabled, RP_MATCHES_RCVD=-2.023, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO) X-UiO-Scanned: C1E0D43C16D4E18FA3AA3F4EF4FCD48C2811D16C X-UiO-SPAM-Test: remote_host: 129.240.6.26 spam_score: -69 maxlevel 80 minaction 0 bait 0 mail/h: 3 total 1310 max/h 10 blacklist 0 greylist 0 ratelimit 0 X-Mailman-Approved-At: Tue, 22 Nov 2011 08:49:56 -0800 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2011 22:35:40 -0000 This weekend up updated my notmuch version to the 0.10 rc2 version available in Debian/unstable, and made a few observations I would like to share. I rebuilt the source on Debian/Squeeze to get it working for the machine I use to read email, and patched it to get the unsorted dump before building it. The reason is that my mail store contain slightly more than 1.2 million emails at the moment, and just dumping the tags take hours without the unsorted dump patch. With the patch it took 40 minutes. After dumping the tags, I moved away the .notmuch index and started reindexing using 'notmuch new'. The indexing took 36 hours. At the start it claimed it would take 10 hours, and it continued to underestimate the amount of time left until the very end. It claimed to have 1 hour left when I checked before I went to bed, and claimed to have 15 minutes left when I woke up 6-7 hours later. Shortly before the indexing finished, the notmuch process was using 1.2 GiB of resident memory according to top. Is the process leaking memory? Running 'notmuch restore' to get my tags back took 106 minutes, and I was very surprised that the restore could not load all the tags stored by 'notmuch dump'. The restore complained about this line: NO*TELEMAX**NORWAYII M0018001012307699038 (unread usit year-2002) The message in question is a bounce from some X400 mail system, and its message id look like this: Message-Id: I would like 'notmuch new' to use less memory and be better at estimating the time left, and would also like 'notmuch restore' to always be able to load the output from 'notmuch dump'. -- Happy hacking Petter Reinholdtsen