Re: nbook: a notmuch based address book written in python

diff --git a/69/c8633ef0cc4ca6cfeb0acd202e131b5d86d8bf b/69/c8633ef0cc4ca6cfeb0acd202e131b5d86d8bf
new file mode 100644 (file)
index 0000000..e32a86e
--- /dev/null
+++ b/69/c8633ef0cc4ca6cfeb0acd202e131b5d86d8bf
@@ -0,0 +1,250 @@
+Return-Path: <teythoon@jade-hamburg.de>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id EFD9F431FAF\r
+       for <notmuch@notmuchmail.org>; Mon, 15 Oct 2012 03:58:44 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id OtvRu9295bwU for <notmuch@notmuchmail.org>;\r
+       Mon, 15 Oct 2012 03:58:43 -0700 (PDT)\r
+Received: from mail.cryptobitch.de (cryptobitch.de [88.198.7.68])\r
+       (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id B80E1431FAE\r
+       for <notmuch@notmuchmail.org>; Mon, 15 Oct 2012 03:58:42 -0700 (PDT)\r
+Received: from mail.jade-hamburg.de (mail.jade-hamburg.de [85.183.11.228])\r
+       (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))\r
+       (No client certificate requested)\r
+       by mail.cryptobitch.de (Postfix) with ESMTPSA id 522395AE1DB\r
+       for <notmuch@notmuchmail.org>; Mon, 15 Oct 2012 12:58:38 +0200 (CEST)\r
+Received: by mail.jade-hamburg.de (Postfix, from userid 401)\r
+       id 9B568DF2A4; Mon, 15 Oct 2012 12:58:37 +0200 (CEST)\r
+Received: from thinkbox.jade-hamburg.de (mail.jade-hamburg.de [85.183.11.228])\r
+       (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))\r
+       (No client certificate requested) (Authenticated sender: teythoon)\r
+       by mail.jade-hamburg.de (Postfix) with ESMTPSA id DDD50DF2A1;\r
+       Mon, 15 Oct 2012 12:58:32 +0200 (CEST)\r
+Received: from teythoon by thinkbox.jade-hamburg.de with local (Exim 4.80)\r
+       (envelope-from <teythoon@thinkbox.jade-hamburg.de>)\r
+       id 1TNiNT-0001d2-79; Mon, 15 Oct 2012 12:58:31 +0200\r
+Content-Type: text/plain; charset="utf-8"\r
+MIME-Version: 1.0\r
+Content-Transfer-Encoding: quoted-printable\r
+From: Justus Winter <4winter@informatik.uni-hamburg.de>\r
+User-Agent: alot/0.3.3+\r
+To: Patrick Totzke <patricktotzke@gmail.com>,\r
+       Suvayu Ali <fatkasuvayu+linux@gmail.com>, notmuch@notmuchmail.org\r
+References: <20120924082646.GA10577@kuru.dyndns-at-home.com>\r
+       <20120925104457.12264.30350@megatron>\r
+       <20121008093429.GC4534@kuru.dyndns-at-home.com>\r
+       <20121013165851.29671.29869@brick.lan>\r
+In-Reply-To: <20121013165851.29671.29869@brick.lan>\r
+Message-ID: <20121015105830.12412.43278@thinkbox.jade-hamburg.de>\r
+Subject: Re: nbook: a notmuch based address book written in python\r
+Date: Mon, 15 Oct 2012 12:58:30 +0200\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 15 Oct 2012 10:58:45 -0000\r
+\r
+Hi Suvayu :)\r
+\r
+welcome to notmuch and python.\r
+\r
+Quoting Patrick Totzke (2012-10-13 18:58:51)\r
+> > > And If I look for my own name, this takes over a minute,\r
+> > > eventually dying. This could be an issue with libnotmuch though.\r
+> > > Possibly, your algorithm takes very long and then reads from an initi=\r
+ally\r
+> > > opened Database object again, which was invalidated by concurrent wri=\r
+tes of other processes..\r
+\r
+Hm no, see below.\r
+\r
+> > > -------------------------------\r
+> > > [~] time nbook Patrick                     =\r
+\r
+> > > =\r
+\r
+> > > Error opening /home/pazz/mail/gmail/[Google Mail].All Mail/cur/133068=\r
+2270_0.12958.megatron,U=3D8766,FMD5=3D66ff6a8bc18a8a3ac4b311daa93d358a:2,S:=\r
+ Too many open files\r
+> > > Traceback (most recent call last):\r
+> > >   File "/home/pazz/bin/nbook", line 167, in <module>\r
+> > >   File "/home/pazz/bin/nbook", line 71, in __init__\r
+> > >   File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/message=\r
+.py", line 233, in get_header\r
+> > > notmuch.errors.NullPointerError\r
+> > > Error in sys.excepthook:\r
+> > > Traceback (most recent call last):\r
+> > >   File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line=\r
+ 66, in apport_excepthook\r
+> > > ImportError: No module named fileutils\r
+> > > =\r
+\r
+> > > Original exception was:\r
+> > > Traceback (most recent call last):\r
+> > >   File "/home/pazz/bin/nbook", line 167, in <module>\r
+> > >   File "/home/pazz/bin/nbook", line 71, in __init__\r
+> > >   File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/message=\r
+.py", line 233, in get_header\r
+> > > notmuch.errors.NullPointerError\r
+> > > nbook Patrick  3.20s user 5.47s system 12% cpu 1:11.65 total\r
+> > > ------------------------------------\r
+> > > =\r
+\r
+> > =\r
+\r
+> > Yes someone else pointed this out too.  Again I'm not sure how to\r
+> > proceed here.  I had a quick look at this last week and it seemed to me\r
+> > the limitation comes from within the python bindings for notmuch.  Do\r
+> > you have any ideas?\r
+> =\r
+\r
+> As mentioned before, I think you invalidate the Database object concurren=\r
+tly\r
+> while your long-running algorithm goes through all messages.\r
+> Xapian doesn't handle concurrent access to the index like a normal=E2=84=\r
+=A2 database would.\r
+> This means you are notified by this error that some changes were detected.\r
+> Maybe the error message should be more telling here though. Teythoon?\r
+\r
+The reason for this error is exactly what the error message says, you\r
+are opening to many files. Check out this limit using ulimit -n:\r
+\r
+% ulimit -n\r
+4096\r
+\r
+This problem is subtle. Here is a minimal test case:\r
+\r
+~~~ snip ~~~\r
+import notmuch\r
+\r
+with notmuch.Database() as db:\r
+    query =3D notmuch.Query(db, 'a').search_messages()\r
+    for msg in query:\r
+        msg.get_header('from')\r
+\r
+with notmuch.Database() as db:\r
+    query =3D notmuch.Query(db, 'a').search_messages()\r
+    for msg in list(query):\r
+        msg.get_header('from')\r
+~~~ snap ~~~\r
+\r
+% python test.py\r
+Error opening /home/teythoon/Maildir/.lists.notmuch/cur/1323251462.M53044P1=\r
+8514.thinkbox,S=3D7306,W=3D7466:2,: Too many open files\r
+Traceback (most recent call last):\r
+  File "test.py", line 11, in <module>\r
+    msg.get_header('from')\r
+  File "/home/teythoon/.local/lib/python2.7/site-packages/notmuch/message.p=\r
+y", line 237, in get_header\r
+    raise NullPointerError()\r
+notmuch.errors.NullPointerError\r
+\r
+Observe that it blows up in line 11, the first version works. The only\r
+difference is that the second version creates a list from the notmuch\r
+query. This prevents the garbage collector from collecting the message\r
+objects and thus closing the file handles. So here's your fix:\r
+\r
+~~~ snip ~~~\r
+diff --git a/nbook b/nbook\r
+index 387c71d..b3d4fd6 100755\r
+--- a/nbook\r
++++ b/nbook\r
+@@ -173,7 +173,7 @@ class AddressHeaders(object):\r
+ # Search\r
+ db =3D Database()\r
+ query =3D Query(db, 'from:"{0}" or to:"{0}"'.format(querystr))\r
+-msgs =3D list(query.search_messages())\r
++msgs =3D query.search_messages()\r
+ =\r
+\r
+ addresses =3D AddressHeaders(msgs, querystr)\r
+ print addresses\r
+~~~ snap ~~~\r
+\r
+A few more comments:\r
+\r
+> from notmuch import *\r
+\r
+Please avoid * imports, they prevent tools like pyflakes from checking\r
+whether you accidentally misspelled any identifiers.\r
+\r
+> pyversion =3D float('%d.%d' % (sys.version_info.major, sys.version_info.m=\r
+inor))\r
+> if pyversion < 2.7:\r
+\r
+Converting this to float feels wrong. Consider doing sth like\r
+\r
+if sys.version_info.major > 2 or (sys.version_info.major =3D=3D 2 and sys.v=\r
+ersion_info.minor >=3D 7):\r
+\r
+>     print '`nbook\' needs Python 2.7 or higher for argparse'\r
+\r
+Note that in py3k print is a function and not a statement, so you need\r
+to use braces. Consider dropping this at the beginning of all your\r
+python files to make py2.7 use the new features:\r
+\r
+from __future__ import print_function, absolute_import, unicode_literals\r
+\r
+>     exit(-1)\r
+\r
+exit is not a builtin function. You have to use sys.exit. Tools like\r
+pyflakes can spot this kind of mistakes. Also, sys.exit also accepts a\r
+string as argument which it prints to stderr before exiting with an\r
+error code.\r
+\r
+>         self.__fromhdr__ +=3D ',' + msg.get_header('from')\r
+\r
+Hm, this is somewhat unpythonic. It used to be the case that building\r
+strings this way was a lot slower than building a list and then\r
+joining it on a delimiter of your choice\r
+(i.e. ','.join(from_headers)). This is (was?) because strings are\r
+immutable in python and constantly creating strings just to throw them\r
+away in the next iteration puts a lot of pressure on the memory\r
+management system. Somewhat recent discussion here:\r
+\r
+http://stackoverflow.com/questions/1316887/what-is-the-most-efficient-strin=\r
+g-concatenation-method-in-python\r
+\r
+>     def print_addrs(self, fmtstr=3D'', query=3D''):\r
+>         if '' =3D=3D fmtstr: fmtstr =3D '%s    %s\n'\r
+\r
+Ok, several things here:\r
+\r
+* The comparison looks weird, you are using the string constant as the\r
+  first operand. While this is technically not wrong, it is somewhat\r
+  unpythonic b/c if you read it out loud (''if the empty string is\r
+  equal to fmtstr'') it somewhat bends the 1:1 mapping of the semantic\r
+  of your program and the English sentence. It looks like this c hack\r
+  that is actually unnecessary in python b/c you cannot use the\r
+  assignment operator as a value (except for a=3Db=3Dc=3D0 style\r
+  assignments).\r
+\r
+* Please don't put multiple statements in one line.\r
+\r
+* This can be written shorter and more idiomatic (yay keyword\r
+  arguments):\r
+\r
+    def print_addrs(self, fmtstr=3D'%s    %s\n', query=3D''):\r
+        [...]\r
+\r
+Happy hacking :)\r
+Justus\r