Re: [notmuch] notmuch new: Memory problem

diff --git a/07/f841a2b67b7e161be87e04f4ddd9fb296a3764 b/07/f841a2b67b7e161be87e04f4ddd9fb296a3764
new file mode 100644 (file)
index 0000000..0f86ec9
--- /dev/null
+++ b/07/f841a2b67b7e161be87e04f4ddd9fb296a3764
@@ -0,0 +1,112 @@
+Return-Path: <dominik.epple@googlemail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id CF375431FBF\r
+       for <notmuch@notmuchmail.org>; Wed, 25 Nov 2009 01:39:58 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id x9UwfLK+nXAb for <notmuch@notmuchmail.org>;\r
+       Wed, 25 Nov 2009 01:39:58 -0800 (PST)\r
+Received: from mail-gx0-f221.google.com (mail-gx0-f221.google.com\r
+       [209.85.217.221])\r
+       by olra.theworths.org (Postfix) with ESMTP id E2D78431FAE\r
+       for <notmuch@notmuchmail.org>; Wed, 25 Nov 2009 01:39:57 -0800 (PST)\r
+Received: by gxk21 with SMTP id 21so6432440gxk.10\r
+       for <notmuch@notmuchmail.org>; Wed, 25 Nov 2009 01:39:57 -0800 (PST)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+       d=googlemail.com; s=gamma;\r
+       h=domainkey-signature:mime-version:received:in-reply-to:references\r
+       :date:message-id:subject:from:to:cc:content-type;\r
+       bh=KUHvVXZJ6rJWpnhvTQS5TEcU0i+1Kvf3jQIkyZsbDkI=;\r
+       b=wqPY1npUkCgt8IEKn+j/Rn/hyk9VKykLzrcpFFNJAfBq4aoXifJle0WPuYvfVncwnr\r
+       GQnFIJQjS25gWEZry2NRO/9O7IwAnDQI9qOiv9qSFWyb1xvt3A8w8LBrIe0Fle9sxlDU\r
+       u7AjXOKhLary7auvaGqSCeTorElaJ2I0UMSbQ=\r
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma;\r
+       h=mime-version:in-reply-to:references:date:message-id:subject:from:to\r
+       :cc:content-type;\r
+       b=kewbTHpLXxDvxY97Kg2GGjt3oDJRG+FK5c4iETc6WJKC2/+bRwxz/9iudDW68n6JVK\r
+       twX7uiUjdau3dMD+pgdYdcG9ZAjQqT1z0wNjPvqirrn6VSL+GVXn2VLV7QpCoLk2ixWK\r
+       x+MpIHzneKEvHRBZZ+nVbTdA8FuX2djp4F2/0=\r
+MIME-Version: 1.0\r
+Received: by 10.90.37.8 with SMTP id k8mr2529160agk.19.1259141997508; Wed, 25 \r
+       Nov 2009 01:39:57 -0800 (PST)\r
+In-Reply-To: <123554aa0911230826o11e54d5ckc90e5ae8dab6ffd3@mail.gmail.com>\r
+References: <123554aa0911200056h73def158pb0db64a2a78ed687@mail.gmail.com>\r
+       <87skc8oqyn.fsf@yoom.home.cworth.org>\r
+       <123554aa0911230826o11e54d5ckc90e5ae8dab6ffd3@mail.gmail.com>\r
+Date: Wed, 25 Nov 2009 10:39:57 +0100\r
+Message-ID: <123554aa0911250139l907c4efs60d704dae962c473@mail.gmail.com>\r
+From: Dominik Epple <dominik.epple@googlemail.com>\r
+To: Carl Worth <cworth@cworth.org>\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+Cc: notmuch@notmuchmail.org\r
+Subject: Re: [notmuch] notmuch new: Memory problem\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.12\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 25 Nov 2009 09:39:59 -0000\r
+\r
+Hello,\r
+\r
+I repeated the procedure (mb2md, notmuch new), but before, I saved all\r
+those large emails with backup logs into a separate folder which i\r
+deleted before "notmuch new". Then, "notmoch new" works as expected.\r
+So the problem stems indeed from too many too large files being\r
+present. (I actually found some being as large as 40M, not just 2.4M,\r
+as written in previous mails.)\r
+\r
+Regards\r
+Dominik\r
+\r
+\r
+2009/11/23 Dominik Epple <dominik.epple@googlemail.com>:\r
+> Hi,\r
+>\r
+> 2009/11/20 Carl Worth <cworth@cworth.org>:\r
+>> On Fri, 20 Nov 2009 09:56:50 +0100, Dominik Epple <dominik.epple@googlemail.com> wrote:\r
+>>> Is there a problem with the number of my mails? I currently have over\r
+>>> 40.000 Mails... they live currently in mbox files, I created a Maildir\r
+>>> with mb2md-3.20.pl.\r
+>>\r
+>> I'm suspecting that you have some big files in there, (such as indexes\r
+>> from some other mail program). We had code in notmuch to detect and\r
+>> ignore these, but a recent bug had broken that.\r
+>>\r
+>> I just fixed this code as of the below commit. So please update and try\r
+>> again and let us know if things work any better.\r
+>\r
+> Ok, one of the problems seems to be solved. One can learn from the\r
+> info: output that the code actually ignores non-email data. These\r
+> files are small and fragments of real mail. Obviously the mb2md code\r
+> made errors there.\r
+>\r
+> But I run in a different issue. I have a lot of files in the Maildir\r
+> which contain base64 encoded binary data. (Some remote site sends my\r
+> its daily backup logs.) Those files are all of 2.4 megabyte in size.\r
+> By adding some debug code to notmuch-new.c, I find out that the\r
+> program becomes very slow and consumes a lot of memory when adding\r
+> these files. I just killed it when it consumed 2 GByte again.\r
+>\r
+> So as you suspected, the problem seems to stem from large files. But\r
+> those large files are not indices or stuff like that from different\r
+> mail programs, but they are valid emails which contain a lot of\r
+> (encoded) binary data.\r
+>\r
+> Perhaps we should be able to configure notmuch such that he ignores\r
+> all mails that match specific pattern (like "Subject: Backup logs\r
+> from.*")\r
+>\r
+> Regards\r
+> Dominik\r
+>\r