Re: [notmuch] notmuch new: Memory problem

diff --git a/b0/f51841674efe2c1c2dddabc6a430e6426833ec b/b0/f51841674efe2c1c2dddabc6a430e6426833ec
new file mode 100644 (file)
index 0000000..edba4e7
--- /dev/null
+++ b/b0/f51841674efe2c1c2dddabc6a430e6426833ec
@@ -0,0 +1,80 @@
+Return-Path: <cworth@cworth.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 2470B431FBF;\r
+       Fri, 20 Nov 2009 08:01:30 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id hjGAfZFPtT5T; Fri, 20 Nov 2009 08:01:28 -0800 (PST)\r
+Received: from cworth.org (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 5FFF9431FAE;\r
+       Fri, 20 Nov 2009 08:01:28 -0800 (PST)\r
+From: Carl Worth <cworth@cworth.org>\r
+To: Dominik Epple <dominik.epple@googlemail.com>, notmuch@notmuchmail.org\r
+In-Reply-To: <123554aa0911200056h73def158pb0db64a2a78ed687@mail.gmail.com>\r
+References: <123554aa0911200056h73def158pb0db64a2a78ed687@mail.gmail.com>\r
+Date: Fri, 20 Nov 2009 17:01:17 +0100\r
+Message-ID: <877htlmbhu.fsf@yoom.home.cworth.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+Subject: Re: [notmuch] notmuch new: Memory problem\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.12\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Fri, 20 Nov 2009 16:01:30 -0000\r
+\r
+On Fri, 20 Nov 2009 09:56:50 +0100, Dominik Epple <dominik.epple@googlemail.com> wrote:\r
+> I am strongly interested in giving notmuch a try.\r
+\r
+Welcome to notmuch, Dominik! I'm sorry your initial attempt to use it\r
+hasn't been quite as smooth as we might like.\r
+\r
+>                                                   But I fail setting\r
+> it up. The problem is that during "notmuch new", memory consumption\r
+> and system load increases to values that make my system unusable. I\r
+> then killed "notmuch new" at a memory consumption of 2.7G and at a\r
+> system load of 7.\r
+\r
+Yikes. That really sounds like something ran out of control consuming\r
+memory. I certainly haven't seen anything like that before.\r
+\r
+> After hitting Ctrl-C, it says "Stopping" but does not stop. I then\r
+> killed "notmuch new" after some minutes with signal KILL.\r
+\r
+After "Stopping" gets printed, the notmuch code won't be doing any more\r
+work. It is expected that it will take some time after that message is\r
+printed before notmuch will actually exit. The extra time is to wait for\r
+Xapian to flush out to disk data that notmuch has already provided to\r
+it.\r
+\r
+I'm curious how big your .notmuch directory ended up after this\r
+operation. (And how that compares in size to the total size of your\r
+collection of mail.)\r
+\r
+> Is there a problem with the number of my mails? I currently have over\r
+> 40.000 Mails... they live currently in mbox files, I created a Maildir\r
+> with mb2md-3.20.pl.\r
+\r
+That's definitely not too much mail. I think you should expect "notmuch\r
+new" currently to index on the order of 10 - 100 messages/sec.\r
+\r
+Your "notmuch new" process should have been reporting a count once per\r
+second as it progressed, (at least until things went wrong). How far did\r
+you see that go?\r
+\r
+I'm wondering if there's a particular file (or files) that are\r
+triggering the bad behavior. Maybe we need a debug option for "notmuch\r
+new" to print the filenames of messages as they are being processed.\r
+\r
+-Carl\r