From: Tomi Ollila Date: Thu, 30 May 2013 16:06:42 +0000 (+0300) Subject: Re: [PATCH] Fix shared library loading in Python bindings. X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=77beffe974a5347c4ca15ad1dfb1945321a65f39;p=notmuch-archives.git Re: [PATCH] Fix shared library loading in Python bindings. --- diff --git a/f1/ead7154ca3e523f0c117fdc15630e13535e683 b/f1/ead7154ca3e523f0c117fdc15630e13535e683 new file mode 100644 index 000000000..0e4c1e60a --- /dev/null +++ b/f1/ead7154ca3e523f0c117fdc15630e13535e683 @@ -0,0 +1,223 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 99689431FBC + for ; Thu, 30 May 2013 09:06:55 -0700 (PDT) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id S8n+lLFKrtT0 for ; + Thu, 30 May 2013 09:06:47 -0700 (PDT) +Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34]) + by olra.theworths.org (Postfix) with ESMTP id 85B7D431FAE + for ; Thu, 30 May 2013 09:06:47 -0700 (PDT) +Received: from guru.guru-group.fi (localhost [IPv6:::1]) + by guru.guru-group.fi (Postfix) with ESMTP id 475A8100093; + Thu, 30 May 2013 19:06:43 +0300 (EEST) +From: Tomi Ollila +To: Justus Winter <4winter@informatik.uni-hamburg.de>, + Julian Berman , notmuch@notmuchmail.org +Subject: Re: [PATCH] Fix shared library loading in Python bindings. +In-Reply-To: <20130530122608.20099.26290@thinkbox.jade-hamburg.de> +References: <1369540418-94177-1-git-send-email-Julian@GrayVines.com> + <20130530122608.20099.26290@thinkbox.jade-hamburg.de> +User-Agent: Notmuch/0.15.2+115~g12cf6af (http://notmuchmail.org) Emacs/24.3.1 + (x86_64-unknown-linux-gnu) +X-Face: HhBM'cA~ +MIME-Version: 1.0 +Content-Type: text/plain +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 30 May 2013 16:06:55 -0000 + +On Thu, May 30 2013, Justus Winter <4winter@informatik.uni-hamburg.de> wrote: + +> Hi Julian :) +> +> Quoting Julian Berman (2013-05-26 05:53:38) +>> Specifically, fixes loading on OS X, where libnotmuch will be +>> a dylib.: +>> --- +>> bindings/python/notmuch/globals.py | 3 ++- +>> 1 file changed, 2 insertions(+), 1 deletion(-) +>> +>> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py +>> index c7632c3..5e08e73 100644 +>> --- a/bindings/python/notmuch/globals.py +>> +++ b/bindings/python/notmuch/globals.py +>> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth +>> """ +>> +>> from ctypes import CDLL, Structure, POINTER +>> +from ctypes.util import find_library +>> +>> #----------------------------------------------------------------------------- +>> #package-global instance of the notmuch library +>> try: +>> - nmlib = CDLL("libnotmuch.so.3") +>> + nmlib = CDLL(find_library("libnotmuch")) +> +> Does this work for you on Darwin? On my box (Debian/Linux) I have to +> use "notmuch" instead of "libnotmuch" to get anything from +> find_library: +> +> In [5]: print ctypes.util.find_library("notmuch") +> libnotmuch.so.3 +> +> In [6]: print ctypes.util.find_library("libnotmuch") +> None +> +> Then again, find_library is different for every architecture under the +> sun... +> +>> except: +>> raise ImportError("Could not find shared 'notmuch' library.") +> +> I'm leaning towards declining the patch in it's current form. If the +> bindings do not work on OS X, we need to find another solution. There +> are two reasons for this: +> +> 1. find_library was once used, but was removed since it is (has?) been +> problematic wrt to LD_LIBRARY_PATH usage: +> +> % git show f378f458 +> commit f378f45893bb4263402008b2abd8aab522901fb6 +> Author: Cedric Cabessa +> Date: Mon Apr 5 03:03:51 2010 +0200 +> +> find_library does not read LD_LIBRARY_PATH, but CDLL does. +> +> diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py +> index ef2686f..fa20ae8 100644 +> --- a/cnotmuch/globals.py +> +++ b/cnotmuch/globals.py +> @@ -3,17 +3,17 @@ from ctypes.util import find_library +> +> #----------------------------------------------------------------------------- +> #package-global instance of the notmuch library +> -#TODO: lazy load this on first access? +> -so = find_library('notmuch') +> -if so is None: +> - raise ImportError("Could not find shared 'notmuch' library.") +> -nmlib = CDLL(so) +> +try: +> + nmlib = CDLL("libnotmuch.so.1") +> +except: +> + raise ImportError("Could not find shared 'notmuch' library.") +> [...] +> +> As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried +> to test whether find_library now respects LD_LIBRARY_PATH or not: +> +> teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH +> /home/teythoon/.local/lib +> teythoon@thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch +> open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3 +> +> That's how it's done now and indeed it finds my libnotmuch. +> +> teythoon@thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch +> +> Nothing. Funny. Let's see: +> +> teythoon@thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")' +> execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0 +> clone(Process 12000 attached +> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000 +> [pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0 +> [pid 12000] clone(Process 12001 attached +> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001 +> [pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0 +> Process 12001 detached +> [pid 12000] --- SIGCHLD (Child exited) @ 0 (0) --- +> Process 12000 detached +> --- SIGCHLD (Child exited) @ 0 (0) --- +> libnotmuch.so.3 +> +> So it also prints libnotmuch.so.3, but only because the version +> installed from the Debian archive is also libnotmuch.so.3: +> +> teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch +> libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3 +> +> So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p +> prints the cache), find_library would have returned libnotmuch.so.2 +> and not my libnotmuch.so.3. So your patch would most likely break this +> kind of setup. +> +> 2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I +> almost got eye cancer from just looking briefly at it... The +> implementation of find_library differs for various operating systems +> and most (all?) of them use fork/exec some program (even gcc in some +> cases) to look up the libraries. I'd like to avoid this if possible, +> maybe even if this means handling OS X as a special case in our +> bindings. For reference, our current solution does not require any +> fork/exec calls: +> +> teythoon@thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' +> execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0 +> +> Thoughts anyone? + +Tried this: + +>>> from ctypes import CDLL +>>> x = CDLL("foobnar.so") +Traceback (most recent call last): + File "", line 1, in + File "/usr/lib/python2.6/ctypes/__init__.py", line 353, in __init__ + self._handle = _dlopen(self._name, mode) +OSError: foobnar.so: cannot open shared object file: No such file or +directory + +Looked into /usr/lib/python2.6/ctypes/__init__.py -- _dlopen() uses +dlopen() from _ctypes module, which is in +/usr/lib/python2.6/lib-dynload/_ctypes.so + +I'd guess this does pretty much the same as dlopen(3) does in Linux +(or Mac -- whatever it does there) + +Could something like + +try: + nmlib = CDLL("libnotmuch.so.3") +except OSError: + try: + nmlib = CDLL("something_macosx_understands") + except: + ImportError("Could not find shared 'notmuch' library.") + + +be used ? + +Just before sending this email I came up with this page: + +https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html + +The 'SEARCHING' section has quite a few words how the file is found. + +> Cheers, +> Justus + + +Tomi