Re: [notmuch] notmuch new: Memory problem

diff --git a/9e/7daa63dd58547b9a57fc114b20293ed9808c76 b/9e/7daa63dd58547b9a57fc114b20293ed9808c76
new file mode 100644 (file)
index 0000000..87f7b37
--- /dev/null
+++ b/9e/7daa63dd58547b9a57fc114b20293ed9808c76
@@ -0,0 +1,96 @@
+Return-Path: <dominik.epple@googlemail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 594A7431FBF\r
+       for <notmuch@notmuchmail.org>; Mon, 23 Nov 2009 08:26:42 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id RUXZIT2QX6To for <notmuch@notmuchmail.org>;\r
+       Mon, 23 Nov 2009 08:26:41 -0800 (PST)\r
+Received: from mail-gx0-f221.google.com (mail-gx0-f221.google.com\r
+       [209.85.217.221])\r
+       by olra.theworths.org (Postfix) with ESMTP id 96046431FAE\r
+       for <notmuch@notmuchmail.org>; Mon, 23 Nov 2009 08:26:41 -0800 (PST)\r
+Received: by gxk21 with SMTP id 21so4801871gxk.10\r
+       for <notmuch@notmuchmail.org>; Mon, 23 Nov 2009 08:26:41 -0800 (PST)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+       d=googlemail.com; s=gamma;\r
+       h=domainkey-signature:mime-version:received:in-reply-to:references\r
+       :date:message-id:subject:from:to:cc:content-type;\r
+       bh=8AFa7rDAzE+o0TZXqMCJIBtlzQtVfiEIkzPdpHsUOcs=;\r
+       b=rv44UcNPFj4APsNwDaJjjUXiDJT6z9ENWhmtyCVeH6ZkhCR6whmnafcqQAnJ3J3opZ\r
+       Sj9opl5JKVXnpoFuZWq2o+3tbeFNYVnPCdLjHwphuE5fWwk9DKzHqqOvbKDZyfTbIH7K\r
+       nS9cNXZZGYFfjdfcLcPcT05RZlnsHLdlPTE78=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma;\r
+       h=mime-version:in-reply-to:references:date:message-id:subject:from:to\r
+       :cc:content-type;\r
+       b=TstgJ6bOxTVhyjX5+Rz/Hfu0c46PpBerJvVcOV2OKD2SwmdstWHbyULy1067fXK/FI\r
+       a7hAgdyyCjrBpXkrwoxE9TStzLeVPRz1Tk9PoAP19W+b6Yard5M6LJvJD5PWKFMBWuo1\r
+       DEu62k99FNjOrdt0miNKzFzCmvFGvHWIYpVFI=\r
+MIME-Version: 1.0\r
+Received: by 10.90.150.17 with SMTP id x17mr1842627agd.57.1258993601174; Mon, \r
+       23 Nov 2009 08:26:41 -0800 (PST)\r
+In-Reply-To: <87skc8oqyn.fsf@yoom.home.cworth.org>\r
+References: <123554aa0911200056h73def158pb0db64a2a78ed687@mail.gmail.com>\r
+       <87skc8oqyn.fsf@yoom.home.cworth.org>\r
+Date: Mon, 23 Nov 2009 17:26:41 +0100\r
+Message-ID: <123554aa0911230826o11e54d5ckc90e5ae8dab6ffd3@mail.gmail.com>\r
+From: Dominik Epple <dominik.epple@googlemail.com>\r
+To: Carl Worth <cworth@cworth.org>\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+Cc: notmuch@notmuchmail.org\r
+Subject: Re: [notmuch] notmuch new: Memory problem\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.12\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 23 Nov 2009 16:26:42 -0000\r
+\r
+Hi,\r
+\r
+2009/11/20 Carl Worth <cworth@cworth.org>:\r
+> On Fri, 20 Nov 2009 09:56:50 +0100, Dominik Epple <dominik.epple@googlemail.com> wrote:\r
+>> Is there a problem with the number of my mails? I currently have over\r
+>> 40.000 Mails... they live currently in mbox files, I created a Maildir\r
+>> with mb2md-3.20.pl.\r
+>\r
+> I'm suspecting that you have some big files in there, (such as indexes\r
+> from some other mail program). We had code in notmuch to detect and\r
+> ignore these, but a recent bug had broken that.\r
+>\r
+> I just fixed this code as of the below commit. So please update and try\r
+> again and let us know if things work any better.\r
+\r
+Ok, one of the problems seems to be solved. One can learn from the\r
+info: output that the code actually ignores non-email data. These\r
+files are small and fragments of real mail. Obviously the mb2md code\r
+made errors there.\r
+\r
+But I run in a different issue. I have a lot of files in the Maildir\r
+which contain base64 encoded binary data. (Some remote site sends my\r
+its daily backup logs.) Those files are all of 2.4 megabyte in size.\r
+By adding some debug code to notmuch-new.c, I find out that the\r
+program becomes very slow and consumes a lot of memory when adding\r
+these files. I just killed it when it consumed 2 GByte again.\r
+\r
+So as you suspected, the problem seems to stem from large files. But\r
+those large files are not indices or stuff like that from different\r
+mail programs, but they are valid emails which contain a lot of\r
+(encoded) binary data.\r
+\r
+Perhaps we should be able to configure notmuch such that he ignores\r
+all mails that match specific pattern (like "Subject: Backup logs\r
+from.*")\r
+\r
+Regards\r
+Dominik\r