From: Antoine Beaupré Date: Mon, 1 Feb 2016 23:31:34 +0000 (+1900) Subject: muchsync runs out of memory X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=013163cecdc76d2e949a7af89a7668cdb9428103;p=notmuch-archives.git muchsync runs out of memory --- diff --git a/6e/0f5215c3dd26d26eadea2d66fa171a71b38851 b/6e/0f5215c3dd26d26eadea2d66fa171a71b38851 new file mode 100644 index 000000000..d4fbe92d4 --- /dev/null +++ b/6e/0f5215c3dd26d26eadea2d66fa171a71b38851 @@ -0,0 +1,185 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by arlo.cworth.org (Postfix) with ESMTP id BE5D46DE099D + for ; Mon, 1 Feb 2016 15:38:04 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at cworth.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from arlo.cworth.org ([127.0.0.1]) + by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id J3c6CosZGBy7 for ; + Mon, 1 Feb 2016 15:38:02 -0800 (PST) +X-Greylist: delayed 383 seconds by postgrey-1.35 at arlo; + Mon, 01 Feb 2016 15:38:02 PST +Received: from marcos.anarc.at (marcos.anarc.at [206.248.172.91]) + by arlo.cworth.org (Postfix) with ESMTP id A824E6DE0222 + for ; Mon, 1 Feb 2016 15:38:02 -0800 (PST) +Received: by marcos.anarc.at (Postfix, from userid 1000) + id 7BA301A008D; Mon, 1 Feb 2016 18:31:34 -0500 (EST) +From: Antoine =?utf-8?Q?Beaupr=C3=A9?= +To: notmuch@notmuchmail.org +Subject: muchsync runs out of memory +User-Agent: Notmuch/0.18.2 (http://notmuchmail.org) Emacs/24.4.1 + (x86_64-pc-linux-gnu) +Date: Mon, 01 Feb 2016 18:31:34 -0500 +Message-ID: <87d1sgm2i1.fsf@marcos.anarc.at> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Transfer-Encoding: quoted-printable +X-Mailman-Approved-At: Fri, 05 Feb 2016 11:46:07 -0800 +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.20 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Mon, 01 Feb 2016 23:38:04 -0000 + +I have been impressed with the features of muchsync when I recently +found out about it. muchsync seems to do everything i want: i have my +server where i usually take my mail, but then my laptop that i want to +sync and run notmuch + emacs on... + +however, when i tried it, it failed: + +[1011]anarcat@angela:~$ muchsync -v --noup --init Maildir +marcos.anarc.at +[SERVER] [notmuch] Note: Ignoring non-mail file: +/home/anarcat/Maildir//.Archives/dovecot-uidlist +[SERVER] [notmuch] Note: Ignoring non-mail file: +/home/anarcat/Maildir//.Archives/dovecot.index.log +[SERVER] [notmuch] Note: Ignoring non-mail file: +/home/anarcat/Maildir//.Drafts/dovecot-uidlist +[...] +[SERVER] [notmuch] Note: Ignoring non-mail file: +/home/anarcat/Maildir//courierimaphieracl/notmuch +[SERVER] [notmuch] Note: Ignoring non-mail file: +/home/anarcat/Maildir//courierimapkeywords/:list +[SERVER] [notmuch] Processed 53 total files in almost no time. +[SERVER] [notmuch] Added 4 new messages to the database. +[SERVER] [notmuch] Tagging spam (still incomplete) +[SERVER] [notmuch] tagging folders +[SERVER] [notmuch] tagging feeds +[SERVER] [notmuch] Tagging the rest +[notmuch] No new mail. +synchronizing muchsync database with Xapian... 4.525728 (+4.525728) +starting scan of Xapian database... 4.526494 (+0.000766) +opened Xapian... 4.530846 (+0.004351) +scanned message IDs... 4.531486 (+0.000640) +scanned tags... 4.532962 (+0.001477) +scanned directories in xapian... 4.533484 (+0.000521) +scanned filenames in xapian... 4.534243 (+0.000760) +adjusted link counts... 4.534790 (+0.000546) +finished synchronizing muchsync database with Xapian... 4.585857 +(+0.051067) +received server's version vector... 4.673185 (+0.087328) +received hashes of new files... 13.704394 (+9.031209) +created directory /home/anarcat/Maildir/.Archives.2010 +created directory /home/anarcat/Maildir/.Archives.2010/cur +created directory /home/anarcat/Maildir/.Archives.2010/new +created directory /home/anarcat/Maildir/.Archives.2010/tmp +created directory /home/anarcat/Maildir/.ham +created directory /home/anarcat/Maildir/.ham/cur +created directory /home/anarcat/Maildir/.ham/new +created directory /home/anarcat/Maildir/.ham/tmp +created directory /home/anarcat/Maildir/.Archives.2011 +created directory /home/anarcat/Maildir/.Archives.2011/cur +created directory /home/anarcat/Maildir/.Archives.2011/new +created directory /home/anarcat/Maildir/.Archives.2011/tmp +Warning: +/home/anarcat/Maildir/.Archives.2011/cur/1447185197.M640040073P4037Q12085Rc= +1c3db4c.angela:2, +is an mbox containing a single message, +likely caused by misconfigured mail delivery. Support for +single-message +mboxes is deprecated and may be removed in the future. +A Xapian exception occurred opening database: Unable to get write lock +on /home/anarcat/Maildir/.notmuch/xapian: Couldn't fork: Cannot allocate +memory +/home/anarcat/Maildir: A Xapian exception occurred +Processus arr=C3=AAt=C3=A9 + +when i run it again, it thinks everything is fine: + +[1012]anarcat@angela:~137$ muchsync -v --noup Maildir marcos.anarc.at +[notmuch] Processed 1 file in almost no time. +[notmuch] Added 1 new message to the database. +synchronizing muchsync database with Xapian... 1.304332 (+1.304332) +starting scan of Xapian database... 1.306140 (+0.001807) +opened Xapian... 1.310448 (+0.004308) +scanned message IDs... 1.656520 (+0.346072) +scanned tags... 1.744574 (+0.088054) +scanned directories in xapian... 1.746279 (+0.001704) +scanned filenames in xapian... 20.900691 (+19.154412) +adjusted link counts... 20.939274 (+0.038584) +finished synchronizing muchsync database with Xapian... 21.248633 +(+0.309359) + +but it's not fine: + +[1019]anarcat@angela:~$ notmuch count +6289 +[1021]anarcat@angela:~$ ssh anarc.at notmuch count +202094 + +at first, it did not look like it used up all the RAM, because +I was looking at Munin graphs which are updated every 5 minutes.. + +after talking with the author privately (the above is a copy of a +previous conversation), I was told to try the following patch + +--- a/muchsync.cc ++++ b/muchsync.cc +@@ -15,7 +15,7 @@ + + using namespace std; + +-#if 0 ++#if 1 + // This gives core dumps to make it easier to debug + struct no_such_exception_t { + const char *what() noexcept { return "no such exception"; } + +Then I could confirm muchsync was killed by the OOM killer: + +In the end, it did run out of memory: + +[47386.773767] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapen= +ts oom_score_adj name +[...] +[47386.774287] [15162] 1000 15162 1280802 835232 2478 8 405017= + 0 muchsync +[...] +[47386.774303] Out of memory: Kill process 15162 (muchsync) score 866 or sa= +crifice child + +... so around 1.2GB in use... I don't think it dumped core - if so, I +couldn't find the core file. + +Note that only 500MB of mails were transfered, out of around 11GB of +notmuch db + mail in my remote maildir. + +Anyone using muchsync with similar data sizes? + +I'd love to get this working - I had trouble reseting my offlineimap + +local notmuch synchronisation the last time I tried... Notmuch tags +would not get assigned properly, for some reason... + +Thanks for any advice, + +a. + +--=20 +Nothing incites to money-crimes like great poverty or great wealth. + - Mark Twain