Re: notmuch killed due to out of memory - how to move forward

diff --git a/06/aba6f513d21523fc0fbd2151e15acbe1c97ea9 b/06/aba6f513d21523fc0fbd2151e15acbe1c97ea9
new file mode 100644 (file)
index 0000000..2395a7c
--- /dev/null
+++ b/06/aba6f513d21523fc0fbd2151e15acbe1c97ea9
@@ -0,0 +1,110 @@
+Return-Path: <jani@nikula.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 1B506431FBC\r
+       for <notmuch@notmuchmail.org>; Tue,  4 Feb 2014 00:25:37 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+       tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id dvf8Y2BBuakD for <notmuch@notmuchmail.org>;\r
+       Tue,  4 Feb 2014 00:25:29 -0800 (PST)\r
+Received: from mail-qa0-f47.google.com (mail-qa0-f47.google.com\r
+       [209.85.216.47]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id C37D8431FAF\r
+       for <notmuch@notmuchmail.org>; Tue,  4 Feb 2014 00:25:29 -0800 (PST)\r
+Received: by mail-qa0-f47.google.com with SMTP id j5so11545887qaq.6\r
+       for <notmuch@notmuchmail.org>; Tue, 04 Feb 2014 00:25:28 -0800 (PST)\r
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+       d=1e100.net; s=20130820;\r
+       h=x-gm-message-state:from:to:subject:in-reply-to:references\r
+       :user-agent:date:message-id:mime-version:content-type;\r
+       bh=a01g9Gmh+8b36jvtepoObCJuCt0oG8P/EvAbxZt5vGM=;\r
+       b=Q4eAFKSbIKoqCMv+8bQz5EW12rDYskTdY+4aT/iqryXqqPLpMJZfozsBVXwbWYNTBy\r
+       XxuuRz4I8iGa1x36CSl02rttCV6ZYg3fYQH8QId3vYRkAgkOxu1KV5anhobzMywWxxGg\r
+       XpnEurnNPko3d8IDjY2R9bGAu0FZu4FHZb4uCoFwRFKIKqi8jGl+QChEKYaLQ2rr9/PU\r
+       OHml6xH3aCmyXuxPsYw/CSzDWXVSzeiTG3yHNCdg9fdFkLEr04Sf1ovAsT4f6gu//pDw\r
+       0kd4ClUxu+WhvhD5QBb9xHg/cO2SBYSM7YNwS8uFGzP+u6/XMDQzlPLFdmTLSzJ0b+xb\r
+       PuPg==\r
+X-Gm-Message-State:\r
+ ALoCoQn7Xs1IuzaYF4174TOpa9o+oE6D3yuFmzKjnTW4wC4uDA4a2DxXpof3IqRO3ZHtlRn9rGtO\r
+X-Received: by 10.140.107.53 with SMTP id g50mr60371021qgf.63.1391502328087;\r
+       Tue, 04 Feb 2014 00:25:28 -0800 (PST)\r
+Received: from localhost ([2001:4b98:dc0:43:216:3eff:fe1b:25f3])\r
+       by mx.google.com with ESMTPSA id o68sm31252698qge.8.2014.02.04.00.25.25\r
+       for <multiple recipients>\r
+       (version=TLSv1.1 cipher=RC4-SHA bits=128/128);\r
+       Tue, 04 Feb 2014 00:25:27 -0800 (PST)\r
+From: Jani Nikula <jani@nikula.org>\r
+To: Hamish Downer <hamish@foobacca.co.uk>, notmuch@notmuchmail.org\r
+Subject: Re: notmuch killed due to out of memory - how to move forward\r
+In-Reply-To:\r
+ <CAOxvSbfWu3M=M1PyFrZcUhszjW0rpb0wYOiia+3sSSd9AGqD4w@mail.gmail.com>\r
+References:\r
+ <CAOxvSbfWu3M=M1PyFrZcUhszjW0rpb0wYOiia+3sSSd9AGqD4w@mail.gmail.com>\r
+User-Agent: Notmuch/0.17+64~g6eaa780 (http://notmuchmail.org) Emacs/23.2.1\r
+       (x86_64-pc-linux-gnu)\r
+Date: Tue, 04 Feb 2014 09:25:17 +0100\r
+Message-ID: <87r47jcmj6.fsf@nikula.org>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=us-ascii\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 04 Feb 2014 08:25:37 -0000\r
+\r
+On Mon, 03 Feb 2014, Hamish Downer <hamish@foobacca.co.uk> wrote:\r
+> Hello\r
+>\r
+> I recently deleted almost 30000 old messages from my maildirs, and\r
+> since I did that, notmuch new has not managed to complete.  I have it\r
+> running on a server with 1 GB of RAM, and the output is telling me\r
+>\r
+>     Cleaned up 25515 of 29803 messages (1m 10s remaining).\r
+>\r
+> at the point it is killed.  I have followed it up to this point with\r
+> top and free and I can see memory usage growing to something like\r
+> 700MB (RES) at which point the process is killed.\r
+>\r
+> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on\r
+> debian wheezy 64 bit.\r
+>\r
+> Any pointers as to what to do to get past this point would be greatly\r
+> appreciated.  If I have just deleted that many messages should I just\r
+> expect notmuch to use that much memory?  (Unfortunately the company\r
+> supplying the server does not allow swap to be enabled so I can't fudge\r
+> it that way)\r
+>\r
+> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't\r
+> produce a report (though I was unsure if I needed to rebuild notmuch\r
+> before that would work).\r
+>\r
+> Happy to build notmuch from source if this might be an interesting bug\r
+> worth tracking down.\r
+\r
+Please try SIGINT, or ^C, on notmuch new before it gets killed. That\r
+should be handled gracefully, making progress, and letting you chop up\r
+and eventually finish the operation. Please let us know if this helps.\r
+\r
+Needless to say, we shouldn't use that much memory just to delete files\r
+from the index.\r
+\r
+\r
+BR,\r
+Jani.\r
+\r