--- /dev/null
+Return-Path: <anarcat@orangeseeds.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by arlo.cworth.org (Postfix) with ESMTP id BE5D46DE099D\r
+ for <notmuch@notmuchmail.org>; Mon, 1 Feb 2016 15:38:04 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at cworth.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from arlo.cworth.org ([127.0.0.1])\r
+ by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id J3c6CosZGBy7 for <notmuch@notmuchmail.org>;\r
+ Mon, 1 Feb 2016 15:38:02 -0800 (PST)\r
+X-Greylist: delayed 383 seconds by postgrey-1.35 at arlo;\r
+ Mon, 01 Feb 2016 15:38:02 PST\r
+Received: from marcos.anarc.at (marcos.anarc.at [206.248.172.91])\r
+ by arlo.cworth.org (Postfix) with ESMTP id A824E6DE0222\r
+ for <notmuch@notmuchmail.org>; Mon, 1 Feb 2016 15:38:02 -0800 (PST)\r
+Received: by marcos.anarc.at (Postfix, from userid 1000)\r
+ id 7BA301A008D; Mon, 1 Feb 2016 18:31:34 -0500 (EST)\r
+From: Antoine =?utf-8?Q?Beaupr=C3=A9?= <anarcat@orangeseeds.org>\r
+To: notmuch@notmuchmail.org\r
+Subject: muchsync runs out of memory\r
+User-Agent: Notmuch/0.18.2 (http://notmuchmail.org) Emacs/24.4.1\r
+ (x86_64-pc-linux-gnu)\r
+Date: Mon, 01 Feb 2016 18:31:34 -0500\r
+Message-ID: <87d1sgm2i1.fsf@marcos.anarc.at>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain; charset=utf-8\r
+Content-Transfer-Encoding: quoted-printable\r
+X-Mailman-Approved-At: Fri, 05 Feb 2016 11:46:07 -0800\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.20\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 01 Feb 2016 23:38:04 -0000\r
+\r
+I have been impressed with the features of muchsync when I recently\r
+found out about it. muchsync seems to do everything i want: i have my\r
+server where i usually take my mail, but then my laptop that i want to\r
+sync and run notmuch + emacs on...\r
+\r
+however, when i tried it, it failed:\r
+\r
+[1011]anarcat@angela:~$ muchsync -v --noup --init Maildir\r
+marcos.anarc.at\r
+[SERVER] [notmuch] Note: Ignoring non-mail file:\r
+/home/anarcat/Maildir//.Archives/dovecot-uidlist\r
+[SERVER] [notmuch] Note: Ignoring non-mail file:\r
+/home/anarcat/Maildir//.Archives/dovecot.index.log\r
+[SERVER] [notmuch] Note: Ignoring non-mail file:\r
+/home/anarcat/Maildir//.Drafts/dovecot-uidlist\r
+[...]\r
+[SERVER] [notmuch] Note: Ignoring non-mail file:\r
+/home/anarcat/Maildir//courierimaphieracl/notmuch\r
+[SERVER] [notmuch] Note: Ignoring non-mail file:\r
+/home/anarcat/Maildir//courierimapkeywords/:list\r
+[SERVER] [notmuch] Processed 53 total files in almost no time.\r
+[SERVER] [notmuch] Added 4 new messages to the database.\r
+[SERVER] [notmuch] Tagging spam (still incomplete)\r
+[SERVER] [notmuch] tagging folders\r
+[SERVER] [notmuch] tagging feeds\r
+[SERVER] [notmuch] Tagging the rest\r
+[notmuch] No new mail.\r
+synchronizing muchsync database with Xapian... 4.525728 (+4.525728)\r
+starting scan of Xapian database... 4.526494 (+0.000766)\r
+opened Xapian... 4.530846 (+0.004351)\r
+scanned message IDs... 4.531486 (+0.000640)\r
+scanned tags... 4.532962 (+0.001477)\r
+scanned directories in xapian... 4.533484 (+0.000521)\r
+scanned filenames in xapian... 4.534243 (+0.000760)\r
+adjusted link counts... 4.534790 (+0.000546)\r
+finished synchronizing muchsync database with Xapian... 4.585857\r
+(+0.051067)\r
+received server's version vector... 4.673185 (+0.087328)\r
+received hashes of new files... 13.704394 (+9.031209)\r
+created directory /home/anarcat/Maildir/.Archives.2010\r
+created directory /home/anarcat/Maildir/.Archives.2010/cur\r
+created directory /home/anarcat/Maildir/.Archives.2010/new\r
+created directory /home/anarcat/Maildir/.Archives.2010/tmp\r
+created directory /home/anarcat/Maildir/.ham\r
+created directory /home/anarcat/Maildir/.ham/cur\r
+created directory /home/anarcat/Maildir/.ham/new\r
+created directory /home/anarcat/Maildir/.ham/tmp\r
+created directory /home/anarcat/Maildir/.Archives.2011\r
+created directory /home/anarcat/Maildir/.Archives.2011/cur\r
+created directory /home/anarcat/Maildir/.Archives.2011/new\r
+created directory /home/anarcat/Maildir/.Archives.2011/tmp\r
+Warning:\r
+/home/anarcat/Maildir/.Archives.2011/cur/1447185197.M640040073P4037Q12085Rc=\r
+1c3db4c.angela:2,\r
+is an mbox containing a single message,\r
+likely caused by misconfigured mail delivery. Support for\r
+single-message\r
+mboxes is deprecated and may be removed in the future.\r
+A Xapian exception occurred opening database: Unable to get write lock\r
+on /home/anarcat/Maildir/.notmuch/xapian: Couldn't fork: Cannot allocate\r
+memory\r
+/home/anarcat/Maildir: A Xapian exception occurred\r
+Processus arr=C3=AAt=C3=A9\r
+\r
+when i run it again, it thinks everything is fine:\r
+\r
+[1012]anarcat@angela:~137$ muchsync -v --noup Maildir marcos.anarc.at\r
+[notmuch] Processed 1 file in almost no time.\r
+[notmuch] Added 1 new message to the database.\r
+synchronizing muchsync database with Xapian... 1.304332 (+1.304332)\r
+starting scan of Xapian database... 1.306140 (+0.001807)\r
+opened Xapian... 1.310448 (+0.004308)\r
+scanned message IDs... 1.656520 (+0.346072)\r
+scanned tags... 1.744574 (+0.088054)\r
+scanned directories in xapian... 1.746279 (+0.001704)\r
+scanned filenames in xapian... 20.900691 (+19.154412)\r
+adjusted link counts... 20.939274 (+0.038584)\r
+finished synchronizing muchsync database with Xapian... 21.248633\r
+(+0.309359)\r
+\r
+but it's not fine:\r
+\r
+[1019]anarcat@angela:~$ notmuch count\r
+6289\r
+[1021]anarcat@angela:~$ ssh anarc.at notmuch count\r
+202094\r
+\r
+at first, it did not look like it used up all the RAM, because\r
+I was looking at Munin graphs which are updated every 5 minutes..\r
+\r
+after talking with the author privately (the above is a copy of a\r
+previous conversation), I was told to try the following patch\r
+\r
+--- a/muchsync.cc\r
++++ b/muchsync.cc\r
+@@ -15,7 +15,7 @@\r
+\r
+ using namespace std;\r
+\r
+-#if 0\r
++#if 1\r
+ // This gives core dumps to make it easier to debug\r
+ struct no_such_exception_t {\r
+ const char *what() noexcept { return "no such exception"; }\r
+\r
+Then I could confirm muchsync was killed by the OOM killer:\r
+\r
+In the end, it did run out of memory:\r
+\r
+[47386.773767] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapen=\r
+ts oom_score_adj name\r
+[...]\r
+[47386.774287] [15162] 1000 15162 1280802 835232 2478 8 405017=\r
+ 0 muchsync\r
+[...]\r
+[47386.774303] Out of memory: Kill process 15162 (muchsync) score 866 or sa=\r
+crifice child\r
+\r
+... so around 1.2GB in use... I don't think it dumped core - if so, I\r
+couldn't find the core file.\r
+\r
+Note that only 500MB of mails were transfered, out of around 11GB of\r
+notmuch db + mail in my remote maildir.\r
+\r
+Anyone using muchsync with similar data sizes?\r
+\r
+I'd love to get this working - I had trouble reseting my offlineimap +\r
+local notmuch synchronisation the last time I tried... Notmuch tags\r
+would not get assigned properly, for some reason...\r
+\r
+Thanks for any advice,\r
+\r
+a.\r
+\r
+--=20\r
+Nothing incites to money-crimes like great poverty or great wealth.\r
+ - Mark Twain\r
projects / notmuch-archives.git / commitdiff