From: Justus Winter <4winter@informatik.uni-hamburg.de> Date: Thu, 30 May 2013 12:26:08 +0000 (+0200) Subject: Re: [PATCH] Fix shared library loading in Python bindings. X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=72e034c025281dc262342640ca50cd6a54eb0dee;p=notmuch-archives.git Re: [PATCH] Fix shared library loading in Python bindings. --- diff --git a/06/9c51317e3a9f020e3d83b27af957c56afff1f3 b/06/9c51317e3a9f020e3d83b27af957c56afff1f3 new file mode 100644 index 000000000..5e818e8bb --- /dev/null +++ b/06/9c51317e3a9f020e3d83b27af957c56afff1f3 @@ -0,0 +1,240 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 868DB431FAF + for ; Thu, 30 May 2013 05:26:44 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id i+1BCzKjDItp for ; + Thu, 30 May 2013 05:26:36 -0700 (PDT) +Received: from mail.cryptobitch.de (cryptobitch.de [88.198.7.68]) + (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 224E5431FAE + for ; Thu, 30 May 2013 05:26:35 -0700 (PDT) +Received: from mail.jade-hamburg.de (mail.jade-hamburg.de [85.183.11.228]) + (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by mail.cryptobitch.de (Postfix) with ESMTPSA id 1D44F64C63D + for ; Thu, 30 May 2013 14:26:16 +0200 (CEST) +Received: by mail.jade-hamburg.de (Postfix, from userid 401) + id 5FAE6DF28B; Thu, 30 May 2013 14:26:14 +0200 (CEST) +Received: from thinkbox.jade-hamburg.de (cryptobitch.de [88.198.7.68]) + (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) + (No client certificate requested) (Authenticated sender: teythoon) + by mail.jade-hamburg.de (Postfix) with ESMTPSA id 8B725DF28B; + Thu, 30 May 2013 14:26:10 +0200 (CEST) +Received: from teythoon by thinkbox.jade-hamburg.de with local (Exim 4.80) + (envelope-from ) + id 1Ui1vl-0003KD-2H; Thu, 30 May 2013 14:26:09 +0200 +Content-Type: multipart/signed; protocol="application/pgp-signature"; + micalg="pgp-sha256"; boundary="===============4092482392989916922==" +MIME-Version: 1.0 +Content-Disposition: inline +From: Justus Winter <4winter@informatik.uni-hamburg.de> +User-Agent: alot/0.3.4 +To: Julian Berman , notmuch@notmuchmail.org +References: <1369540418-94177-1-git-send-email-Julian@GrayVines.com> +In-Reply-To: <1369540418-94177-1-git-send-email-Julian@GrayVines.com> +Message-ID: <20130530122608.20099.26290@thinkbox.jade-hamburg.de> +Subject: Re: [PATCH] Fix shared library loading in Python bindings. +Date: Thu, 30 May 2013 14:26:08 +0200 +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 30 May 2013 12:26:44 -0000 + +--===============4092482392989916922== +Content-Type: text/plain; charset="utf-8" +MIME-Version: 1.0 +Content-Transfer-Encoding: quoted-printable + +Hi Julian :) + +Quoting Julian Berman (2013-05-26 05:53:38) +> Specifically, fixes loading on OS X, where libnotmuch will be +> a dylib.: +> --- +> bindings/python/notmuch/globals.py | 3 ++- +> 1 file changed, 2 insertions(+), 1 deletion(-) +> = + +> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch= +/globals.py +> index c7632c3..5e08e73 100644 +> --- a/bindings/python/notmuch/globals.py +> +++ b/bindings/python/notmuch/globals.py +> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth +> """ +> = + +> from ctypes import CDLL, Structure, POINTER +> +from ctypes.util import find_library +> = + +> #-----------------------------------------------------------------------= +------ +> #package-global instance of the notmuch library +> try: +> - nmlib =3D CDLL("libnotmuch.so.3") +> + nmlib =3D CDLL(find_library("libnotmuch")) + +Does this work for you on Darwin? On my box (Debian/Linux) I have to +use "notmuch" instead of "libnotmuch" to get anything from +find_library: + +In [5]: print ctypes.util.find_library("notmuch") +libnotmuch.so.3 + +In [6]: print ctypes.util.find_library("libnotmuch") +None + +Then again, find_library is different for every architecture under the +sun... + +> except: +> raise ImportError("Could not find shared 'notmuch' library.") + +I'm leaning towards declining the patch in it's current form. If the +bindings do not work on OS X, we need to find another solution. There +are two reasons for this: + +1. find_library was once used, but was removed since it is (has?) been +problematic wrt to LD_LIBRARY_PATH usage: + +% git show f378f458 +commit f378f45893bb4263402008b2abd8aab522901fb6 +Author: Cedric Cabessa +Date: Mon Apr 5 03:03:51 2010 +0200 + + find_library does not read LD_LIBRARY_PATH, but CDLL does. + +diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py +index ef2686f..fa20ae8 100644 +--- a/cnotmuch/globals.py ++++ b/cnotmuch/globals.py +@@ -3,17 +3,17 @@ from ctypes.util import find_library + = + + #-------------------------------------------------------------------------= +---- + #package-global instance of the notmuch library +-#TODO: lazy load this on first access? +-so =3D find_library('notmuch') +-if so is None: +- raise ImportError("Could not find shared 'notmuch' library.") +-nmlib =3D CDLL(so) ++try: ++ nmlib =3D CDLL("libnotmuch.so.1") ++except: ++ raise ImportError("Could not find shared 'notmuch' library.") +[...] + +As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried +to test whether find_library now respects LD_LIBRARY_PATH or not: + +teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH +/home/teythoon/.local/lib +teythoon@thinkbox ~ % strace -e trace=3Dopen python -c 'import ctypes; ctyp= +es.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch +open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) =3D 3 + +That's how it's done now and indeed it finds my libnotmuch. + +teythoon@thinkbox ~ % strace -f -e trace=3Dstat,open python -c 'import ctyp= +es.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch + +Nothing. Funny. Let's see: + +teythoon@thinkbox ~ % strace -f -e trace=3Dfork,execve,clone python -c 'imp= +ort ctypes.util; print ctypes.util.find_library("notmuch")' +execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctype= +s"...], [/* 63 vars */]) =3D 0 +clone(Process 12000 attached +child_stack=3D0, flags=3DCLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, c= +hild_tidptr=3D0x7f0734a649d0) =3D 12000 +[pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"]= +, [/* 63 vars */]) =3D 0 +[pid 12000] clone(Process 12001 attached +child_stack=3D0, flags=3DCLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, c= +hild_tidptr=3D0x7f119d5479d0) =3D 12001 +[pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars = +*/]) =3D 0 +Process 12001 detached +[pid 12000] --- SIGCHLD (Child exited) @ 0 (0) --- +Process 12000 detached +--- SIGCHLD (Child exited) @ 0 (0) --- +libnotmuch.so.3 + +So it also prints libnotmuch.so.3, but only because the version +installed from the Debian archive is also libnotmuch.so.3: + +teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch + libnotmuch.so.3 (libc6,x86-64) =3D> /usr/lib/libnotmuch.so.3 + +So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p +prints the cache), find_library would have returned libnotmuch.so.2 +and not my libnotmuch.so.3. So your patch would most likely break this +kind of setup. + +2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I +almost got eye cancer from just looking briefly at it... The +implementation of find_library differs for various operating systems +and most (all?) of them use fork/exec some program (even gcc in some +cases) to look up the libraries. I'd like to avoid this if possible, +maybe even if this means handling OS X as a special case in our +bindings. For reference, our current solution does not require any +fork/exec calls: + +teythoon@thinkbox ~ % strace -f -e trace=3Dexecve,fork,clone python -c 'imp= +ort ctypes; ctypes.CDLL("libnotmuch.so.3")' = + +execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"li= +bn"...], [/* 63 vars */]) =3D 0 + +Thoughts anyone? + +Cheers, +Justus + +--===============4092482392989916922== +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Content-Description: signature +Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii" + +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.12 (GNU/Linux) + +iQIcBAABCAAGBQJRp0VcAAoJENMeiILK3jZYJwMP/R4VQYUmEDFYHX7xuiLL4zoC +r+afAVY4OE2u2c9bncT54QairXuDGlLs8wmpDwkfZeiZ1N8OXeEwDlcMZYZ8BpQD +/5LW0Gcf4u91UIsQ6t3wQLj6slvnMEQmLgyWbHpZIjFMbi/sMF7akJ+Ov/eZf0Si +/9ZTrXteoUWw7kw9NhIqyxRC9/WHB9TwSdCVj7QPRBxK78B3+OxoX7DIp40QB3eZ +MPiaOWRqV/6PrEmM1CxIRAEb5EoNB5LdgMM+0Ip6DtoEoKGbt3E7BbBk//tVTtaZ +OYTCDEDtUfv0bfKL/VHVyihFZDJjDN1H+VgJZ4Sai3rjME1k/HphT/ZiBSO8b995 +tpsc3yBB76kWEnHd1wf6DFJTLiKDwhm46V9WeNZiZ9auUm4RfcHozejPIooog9Bo ++qubc9r7FhZpV7qrh9+PfS5MPf7qXb+Q5BpZMU30Cqksy1BwrPtmWJ6dDqucI+pR +/9LxMTxfm8Qub8Fy811nOuG8vcKuL0GxvN/cSM2oqgceDiTXqwCWDO6y3jO3+HBf +yChjoPioIdxfsyRv3UWjxVI0am3lkN0lKHp7tUbCxank9Yaj1yW/DOQhvpt5pNLe +NvuNfYPG+Gk0IIsdOWM+R7EBkkPZmnMGdIJ+fsKPxgFZegeYJRP0FejHtw/HoWhG +R9uuna4ewRUTTFnhxxLU +=IPQn +-----END PGP SIGNATURE----- + +--===============4092482392989916922==--