--- /dev/null
+Return-Path: <tomi.ollila@iki.fi>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 4830A431FAF\r
+ for <notmuch@notmuchmail.org>; Tue, 22 Jan 2013 05:43:21 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id Ek585k-Kxx7q for <notmuch@notmuchmail.org>;\r
+ Tue, 22 Jan 2013 05:43:19 -0800 (PST)\r
+Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])\r
+ by olra.theworths.org (Postfix) with ESMTP id 6A13D431FAE\r
+ for <notmuch@notmuchmail.org>; Tue, 22 Jan 2013 05:43:19 -0800 (PST)\r
+Received: from guru.guru-group.fi (localhost [IPv6:::1])\r
+ by guru.guru-group.fi (Postfix) with ESMTP id 66578100086;\r
+ Tue, 22 Jan 2013 15:43:07 +0200 (EET)\r
+From: Tomi Ollila <tomi.ollila@iki.fi>\r
+To: Jani Nikula <jani@nikula.org>, Mark Walters <markwalters1009@gmail.com>,\r
+ notmuch@notmuchmail.org\r
+Subject: Re: [PATCH 0/5] notmuch batch count\r
+In-Reply-To: <87fw1u30zz.fsf@nikula.org>\r
+References: <cover.1358273133.git.jani@nikula.org> <8738y2ui4y.fsf@qmul.ac.uk>\r
+ <m27gndsotu.fsf@guru.guru-group.fi> <87fw1u30zz.fsf@nikula.org>\r
+User-Agent: Notmuch/0.15+11~ge1e719d (http://notmuchmail.org) Emacs/24.2.1\r
+ (x86_64-unknown-linux-gnu)\r
+X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL\r
+ $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F\r
+ !)g;OY^,BjTbr)Np:%c_o'jj,Z\r
+Date: Tue, 22 Jan 2013 15:43:07 +0200\r
+Message-ID: <m2r4ldjpuc.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 22 Jan 2013 13:43:21 -0000\r
+\r
+On Mon, Jan 21 2013, Jani Nikula <jani@nikula.org> wrote:\r
+\r
+> On Wed, 16 Jan 2013, Tomi Ollila <tomi.ollila@iki.fi> wrote:\r
+>> One could argue that we'd should send json "documents" to notmuch in\r
+>> stdin and notmuch would output json(/sexp) "documents". That is just\r
+>> SMOP. I bet Austin would like this solution, especially the part\r
+>> that involves writing or integrating json parser >;). \r
+>> I'd be happy with this 'batch' approach. \r
+>>\r
+>> I'll be testing this soon, but refrain from reviewing the code\r
+>> until 0.15 is out.\r
+>\r
+> id:87a9s5cp38.fsf@zancas.localnet ;)\r
+\r
+OK :D\r
+\r
+I started testing:\r
+\r
+First a remote script to prepare:\r
+\r
+cat > remote-notmuch-test.sh << EOF\r
+#!/bin/bash\r
+printf -v ARGS '%q ' "$@" # bash feature\r
+ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS\r
+EOF\r
+chmod 755 remote-notmuch-test.sh\r
+\r
+Note 2 things about the script:\r
+\r
+ 1) '~' is in quotes so shell doesn't expand it -- ssh uses /etc/passwd\r
+ instead of environment variable to get HOME (security feature).\r
+ This can be easly tested by prefixing the command with HOME=/tmp.\r
+ 2) The remote address '0.1' is ignored when control socket is active\r
+ if control socket is down, connection will fail (0.0 connects to\r
+ localhost (I'm not sure which interface in that case)).\r
+\r
+Another "script", for emacs:\r
+\r
+cat > remote-notmuch-test.el << EOF\r
+(load "notmuch")\r
+(setq notmuch-command "/home/too/remote-notmuch-test.sh")\r
+\r
+(setq notmuch-saved-searches '(("inbox" . "tag:inbox")\r
+ ("unread" . "tag:unread")\r
+ ("contrib" . "tag:notmuch::contrib")\r
+ ("pushed" . "tag:notmuch::pushed")\r
+ ("needs-review" . "tag:notmuch::needs-review")\r
+ ("moreinfo" . "tag:notmuch::moreinfo")\r
+ ("stale" . "tag:notmuch::stale")\r
+ ("wip" . "tag:notmuch::wip")\r
+ ("bug" . "tag:notmuch::bug")\r
+ ("obsolete" . "tag:notmuch::obsolete")))\r
+\r
+(notmuch)\r
+EOF\r
+\r
+First ssh connection with ControlMaster needs to be started\r
+from one terminal:\r
+\r
+ssh -S '~'/.ssh/master-notmuch-remote1 -M remote.host.tld\r
+\r
+Then, from another terminal, emacs:\r
+\r
+emacs -l remote-notmuch-test.el\r
+\r
+works, takes quite a long time -- opening 'all tags' even longer.\r
+\r
+Let's edit wrapper script:\r
+\r
+cat > remote-notmuch-test.sh << EOF\r
+#!/bin/bash\r
+printf -v ARGS '%q ' "$@" # bash feature\r
+echo /usr/bin/time ./remote-notmuch-test.sh $ARGS >> notmuch-replay.$PPID.log\r
+ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS\r
+EOF\r
+chmod 755 remote-notmuch-test.sh\r
+\r
+Now, after emacs -l remote-notmuch-test.el run and all tags opened,\r
+exit emacs and run the replay script\r
+\r
+env TIME='grepme %Us user %Ss system %P cpu %E total' /usr/bin/time sh ../notmuch-replay.1744.log\r
+last line outputs:\r
+grepme 0.38s user 0.40s system 1% cpu 0:48.21 total\r
+\r
+and wc notmuch-replay.1744.log\r
+ 61 240 3787 notmuch-replay.1744.log\r
+\r
+and each of the individual command in script printed either\r
+\r
+grepme 0.00s user 0.00s system 1% cpu 0:00.78 total or\r
+grepme 0.00s user 0.00s system 1% cpu 0:00.79 total.\r
+\r
+( 0.785 * 61 = 47.9 so notmuch was wasted elsewhere :)\r
+\r
+---\r
+\r
+Ok, now patching 2 notmuch trees, one in local and one in\r
+remote machine and building and installing those.\r
+\r
+Running emacs -l remote-notmuch-test.el \r
+was fast (not in one second, but just a few).\r
+\r
+Unsuprisingly, the "replay" script cannot be run anymore\r
+as it requires input from stdin, therefore tuning it as follows\r
+\r
+cat > remote-notmuch-test.sh << EOF\r
+#!/bin/bash\r
+printf -v ARGS '%q ' "$@" # bash feature\r
+echo `date`: $ARGS >> notmuch-dates.$PPID.log\r
+echo /usr/bin/time ./remote-notmuch-test.sh $ARGS >> notmuch-replay.$PPID.log\r
+ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS\r
+date >> notmuch-dates.$PPID.log\r
+EOF\r
+chmod 755 remote-notmuch-test.sh\r
+\r
+now, again: emacs -l remote-notmuch-test.el \r
+then 'echo output of echo : >> notmuch-dates.2678.log'\r
+and finally opening 'all tags' \r
+\r
+cat notmuch-dates.2678.log\r
+\r
+Tue 22 Jan 15:04:34 EET 2013: count\r
+Tue 22 Jan 15:04:35 EET 2013\r
+Tue 22 Jan 15:04:35 EET 2013: count --batch\r
+Tue 22 Jan 15:04:36 EET 2013\r
+Tue 22 Jan 15:04:36 EET 2013: search-tags\r
+Tue 22 Jan 15:04:37 EET 2013\r
+output of echo :\r
+Tue 22 Jan 15:04:45 EET 2013: count\r
+Tue 22 Jan 15:04:45 EET 2013\r
+Tue 22 Jan 15:04:45 EET 2013: count --batch\r
+Tue 22 Jan 15:04:47 EET 2013\r
+Tue 22 Jan 15:04:47 EET 2013: search-tags\r
+Tue 22 Jan 15:04:47 EET 2013\r
+Tue 22 Jan 15:04:47 EET 2013: count --batch\r
+Tue 22 Jan 15:04:49 EET 2013\r
+\r
+3 seconds for first tags and 4 seconds for second tags (approximately :)\r
+-- quite an improvement 48 seconds.\r
+\r
+Also, changes work fine. Now that I forgot my laptop charger to office\r
+I'll have to wait until tomorrow to do the core review.\r
+\r
+(maybe the 2 'count's could be combined in elisp code ;)\r
+\r
+>\r
+> J.\r
+\r
+Tomi\r
projects / notmuch-archives.git / commitdiff