From: Kushal Kumaran Date: Tue, 4 Feb 2014 09:01:10 +0000 (+0530) Subject: Re: notmuch killed due to out of memory - how to move forward X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=b3a26762b42712eca21a11e7ef5dbdf625bbbd8a;p=notmuch-archives.git Re: notmuch killed due to out of memory - how to move forward --- diff --git a/0c/1977568ba5789353dfbbe18488701c4b073ad0 b/0c/1977568ba5789353dfbbe18488701c4b073ad0 new file mode 100644 index 000000000..bbec25b61 --- /dev/null +++ b/0c/1977568ba5789353dfbbe18488701c4b073ad0 @@ -0,0 +1,145 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id B53DA431FBC + for ; Tue, 4 Feb 2014 01:01:22 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.799 +X-Spam-Level: +X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id 4TtVlK3eIWbN for ; + Tue, 4 Feb 2014 01:01:15 -0800 (PST) +Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com + [209.85.160.172]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 0C8C6431FAF + for ; Tue, 4 Feb 2014 01:01:14 -0800 (PST) +Received: by mail-yk0-f172.google.com with SMTP id 200so20714687ykr.3 + for ; Tue, 04 Feb 2014 01:01:13 -0800 (PST) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; + h=from:to:subject:in-reply-to:references:user-agent:date:message-id + :mime-version:content-type; + bh=AJUu7+gxvVgzYpgIaFMXEFkHVTA8W++gTYZCywtnGRA=; + b=E2Unmpa7yZqWNrp9QN8HKPamgZydFt3zph+/lrCAadzTx1CJ1NqkfJskwiJbNIiOrs + OEqgAI8dCMRFAM0oKBzFHGYrBQggs1UmGPSF4vjtTch2GfdrTRpID+8F3fGbo1XutHir + 0NCXqD6JRlzPahy/mmThia8BCa9e90kWT4n4dQgfJZIC9/99BzsM87CnWKI46azndvfv + 08VrWXG+UQmRyMdBbpvPLDY9ZczUIhXCWybRbYRD7js5bL/+QNbq5w1GSgQTL1tA3PRR + NsCpQmZXoqkI6Hn95Y4fxgUk2wqd4VCBLxRXZHS5CjWpa4dC2OOx2tWqowpZaM6AkBPH + lbMA== +X-Received: by 10.236.151.198 with SMTP id b46mr38141076yhk.3.1391504473164; + Tue, 04 Feb 2014 01:01:13 -0800 (PST) +Received: from magnesium ([198.199.83.138]) by mx.google.com with ESMTPSA id + v41sm20730223yhi.19.2014.02.04.01.01.11 for + (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); + Tue, 04 Feb 2014 01:01:12 -0800 (PST) +Received: by magnesium (Postfix, from userid 1000) + id 330404503E; Tue, 4 Feb 2014 14:31:11 +0530 (IST) +From: Kushal Kumaran +To: notmuch@notmuchmail.org +Subject: Re: notmuch killed due to out of memory - how to move forward +In-Reply-To: <87r47jcmj6.fsf@nikula.org> +References: + + <87r47jcmj6.fsf@nikula.org> +User-Agent: Notmuch/0.17+35~g3b36898 (http://notmuchmail.org) Emacs/24.3.1 + (x86_64-pc-linux-gnu) +Date: Tue, 04 Feb 2014 14:31:10 +0530 +Message-ID: <52f0ac58.4188ec0a.7ff8.ffff9762@mx.google.com> +MIME-Version: 1.0 +Content-Type: multipart/signed; boundary="=-=-="; + micalg=pgp-sha1; protocol="application/pgp-signature" +X-Mailman-Approved-At: Tue, 04 Feb 2014 09:26:02 -0800 +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 04 Feb 2014 09:01:22 -0000 + +--=-=-= +Content-Type: text/plain +Content-Transfer-Encoding: quoted-printable + +Jani Nikula writes: + +> On Mon, 03 Feb 2014, Hamish Downer wrote: +>> Hello +>> +>> I recently deleted almost 30000 old messages from my maildirs, and +>> since I did that, notmuch new has not managed to complete. I have it +>> running on a server with 1 GB of RAM, and the output is telling me +>> +>> Cleaned up 25515 of 29803 messages (1m 10s remaining). +>> +>> at the point it is killed. I have followed it up to this point with +>> top and free and I can see memory usage growing to something like +>> 700MB (RES) at which point the process is killed. +>> +>> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on +>> debian wheezy 64 bit. +>> +>> Any pointers as to what to do to get past this point would be greatly +>> appreciated. If I have just deleted that many messages should I just +>> expect notmuch to use that much memory? (Unfortunately the company +>> supplying the server does not allow swap to be enabled so I can't fudge +>> it that way) +>> +>> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't +>> produce a report (though I was unsure if I needed to rebuild notmuch +>> before that would work). +>> +>> Happy to build notmuch from source if this might be an interesting bug +>> worth tracking down. +> +> Please try SIGINT, or ^C, on notmuch new before it gets killed. That +> should be handled gracefully, making progress, and letting you chop up +> and eventually finish the operation. Please let us know if this helps. +> +> Needless to say, we shouldn't use that much memory just to delete files +> from the index. +> + +When I initially indexed my email archives into notmuch, I had problems +with thermal cutouts, so I did something like this, which automated the +workaround you mention: + +while :; do + timeout -s INT 2m notmuch new + sleep 600 +done + +Of course, with memory issues as the constraint rather than temperature, +the sleep should not be necessary. + +=2D-=20 +regards, +kushal + +--=-=-= +Content-Type: application/pgp-signature + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.14 (GNU/Linux) + +iQEcBAEBAgAGBQJS8KxWAAoJEKolkdITDqLt8bEIALOGQuoW+tiREnDcM4EFJ7FS +OWnEOi6aqxHTQoIY6ADQ3dm1a6p2albi8QY4jMlNVC0ODJiRFd8hExcIkUXlU2hU +cNA8KOKcM4ILEVrzjoies7yJLiibCx/mxRtSyGe8HC78z0j58C/HT9W4cl3fxcQA +k1ImqsVeY56VkOAClPUeUT8jwseZIMIbZTlq735HfkLk+gHwBkrCHeIMdjFf7eWg +PfLQXaEnkq5jNFvG8jVf4JI+BADYr+IOi507yiynMhdtxf5zHTumn2m+qLxw6+tW ++Nnv2NXwSwdTW0Z7HHQYRcTfHZql5x//4tNWzgJyMp/w3a2tljfdcf4JEo6p6CE= +=oqgO +-----END PGP SIGNATURE----- +--=-=-=--