From: Stefan Behnel Date: Tue, 18 Jan 2011 17:38:14 +0000 (+0100) Subject: updated C library tutorial as proposed by Terry Reedy X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=5ebe3fe51320e0c0c3d409490f309ec4fe98e92d;p=cython.git updated C library tutorial as proposed by Terry Reedy --- diff --git a/src/tutorial/clibraries.rst b/src/tutorial/clibraries.rst index bfe32a7a..b9171235 100644 --- a/src/tutorial/clibraries.rst +++ b/src/tutorial/clibraries.rst @@ -2,8 +2,8 @@ Using C libraries ================= Apart from writing fast code, one of the main use cases of Cython is -to call external C libraries from Python code. As seen for the C -string decoding functions above, it is actually trivial to call C +to call external C libraries from Python code. As Cython code +compiles down to C code itself, it is actually trivial to call C functions directly in the code. The following describes what needs to be done to use an external C library in Cython code. @@ -21,6 +21,8 @@ type that can encapsulate all memory management. The C API of the queue implementation, which is defined in the header file ``libcalg/queue.h``, essentially looks like this:: + /* file: queue.h */ + typedef struct _Queue Queue; typedef void *QueueValue; @@ -40,6 +42,8 @@ file ``libcalg/queue.h``, essentially looks like this:: To get started, the first step is to redefine the C API in a ``.pxd`` file, say, ``cqueue.pxd``:: + # file: cqueue.pxd + cdef extern from "libcalg/queue.h": ctypedef struct Queue: pass @@ -59,54 +63,64 @@ file, say, ``cqueue.pxd``:: bint queue_is_empty(Queue* queue) Note how these declarations are almost identical to the header file -declarations, so you can often just copy them over. One exception is -the last line. The return value of the ``queue_is_empty`` method is -actually a C boolean value, i.e. it is either zero or non-zero, -indicating if the queue is empty or not. This is best expressed by -Cython's ``bint`` type, which is a normal ``int`` type when used in C -but maps to Python's boolean values ``True`` and ``False`` when -converted to a Python object. Another difference is the first -line. ``Queue`` is in this case used as an *opaque handle*; only the -library that is called know what is actually inside. Since no Cython code -needs to know the contents of the struct, we do not need to declare its contents, -so we simply provide an empty definition (as we do not want to declare -the ``_Queue`` type which is referenced in the C header) [#]_. +declarations, so you can often just copy them over. One noteworthy +difference is the first line. ``Queue`` is in this case used as an +*opaque handle*; only the library that is called knows what is really +inside. Since no Cython code needs to know the contents of the +struct, we do not need to declare its contents, so we simply provide +an empty definition (as we do not want to declare the ``_Queue`` type +which is referenced in the C header) [#]_. .. [#] There's a subtle difference between ``cdef struct Queue: pass`` - and ``ctypedef struct Queue: pass``. The former declares a type - which is referenced in C code as ``struct Queue``, while the - latter is referenced in C as ``Queue``. This is a C language - quirk that Cython is not able to hide. Most modern C libraries - use the ``ctypedef`` kind of struct. + and ``ctypedef struct Queue: pass``. The former declares a + type which is referenced in C code as ``struct Queue``, while + the latter is referenced in C as ``Queue``. This is a C + language quirk that Cython is not able to hide. Most modern C + libraries use the ``ctypedef`` kind of struct. + +Another exception is the last line. The integer return value of the +``queue_is_empty`` method is actually a C boolean value, i.e. it is +either zero or non-zero, indicating if the queue is empty or not. +This is best expressed by Cython's ``bint`` type, which is a normal +``int`` type when used in C but maps to Python's boolean values +``True`` and ``False`` when converted to a Python object. Next, we need to design the Queue class that should wrap the C queue. +It will live in a file called ``queue.pyx``. [#]_ + +.. [#] Note that the name of the ``.pyx`` file must be different from + the ``cqueue.pxd`` file with declarations from the C library, + as both do not describe the same code. A ``.pxd`` file next to + a ``.pyx`` file with the same name defines exported + declarations for code in the ``.pyx`` file. + Here is a first start for the Queue class:: + # file: queue.pyx + cimport cqueue - cimport python_exc cdef class Queue: cdef cqueue.Queue _c_queue def __cinit__(self): self._c_queue = cqueue.queue_new() -Note that it says ``__cinit__`` rather than ``__init__``. While +Note that it says ``__cinit__`` rather than ``__init__``. While ``__init__`` is available as well, it is not guaranteed to be run (for -instance, one could create a subclass and forget to call the ancestor -constructor). Because not initializing C pointers often leads to -crashing the Python interpreter without leaving as much as a stack -trace, Cython provides ``__cinit__`` which is *always* called on -construction. However, as ``__cinit__`` is called during object -construction, ``self`` is not fully -constructed yet, and one must avoid doing anything with ``self`` but -assigning to ``cdef`` fields. +instance, one could create a subclass and forget to call the +ancestor's constructor). Because not initializing C pointers often +leads to crashing the Python interpreter without leaving as much as a +stack trace, Cython provides ``__cinit__`` which is *always* called on +construction. However, as ``__cinit__`` is called during object +construction, ``self`` is not fully constructed yet, and one must +avoid doing anything with ``self`` but assigning to ``cdef`` fields. Note also that the above method takes no parameters, although subtypes may want to accept some. Although it is guaranteed to get called, the no-arguments ``__cinit__()`` method is a special case here as it does -not prevent subclasses from adding parameters as they see fit. If -parameters are added they must match those of any declared ``__init__`` -method. +not prevent subclasses from adding parameters as they see fit. If +parameters are added they must match those of any declared +``__init__`` method. Before we continue implementing the other methods, it is important to understand that the above implementation is not safe. In case @@ -117,45 +131,67 @@ only reason why the above can fail is due to insufficient memory. In that case, it will return ``NULL``, whereas it would normally return a pointer to the new queue. -The normal way to get out of this is to raise an exception, but -allocating a new exception instance may actually fail when we are -running out of memory. Luckily, CPython provides a function -``PyErr_NoMemory()`` that raises the right exception for us. We can -thus change the init function as follows:: +The normal Python way to get out of this is to raise an exception, but +in this specific case, allocating a new exception instance may +actually fail because we are running out of memory. Luckily, CPython +provides a function ``PyErr_NoMemory()`` that safely raises the right +exception for us. We can thus change the init function as follows:: + cimport cpython.exc # standard cimport from CPython's C-API + cimport cqueue + + cdef class Queue: + cdef cqueue.Queue _c_queue def __cinit__(self): self._c_queue = cqueue.queue_new() if self._c_queue is NULL: - python_exc.PyErr_NoMemory() - -The next thing to do is to clean up when the Queue is no longer used. -To this end, CPython provides a callback that Cython makes available -as a special method ``__dealloc__()``. In our case, all we have to do -is to free the Queue, but only if we succeeded in initialising it in + cpython.exc.PyErr_NoMemory() + +The ``cpython`` package contains pre-defined ``.pxd`` files that ship +with Cython. If you need any CPython C-API functions, you can cimport +them from this package. See Cython's ``Cython/Includes/`` source +package for a complete list of ``.pxd`` files, including parts of the +standard C library. + +The next thing to do is to clean up when the Queue instance is no +longer used (i.e. all references to it have been deleted). To this +end, CPython provides a callback that Cython makes available as a +special method ``__dealloc__()``. In our case, all we have to do is +to free the C Queue, but only if we succeeded in initialising it in the init method:: def __dealloc__(self): if self._c_queue is not NULL: cqueue.queue_free(self._c_queue) -At this point, we have a compilable Cython module that we can test. -To compile it, we need to configure a ``setup.py`` script for -distutils. Based on the example presented earlier on, we can extend -the script to include the necessary setup for building against the -external C library. Assuming it's installed in the normal places -(e.g. under ``/usr/lib`` and ``/usr/include`` on a Unix-like system), -we could simply change the extension setup from +At this point, we have a working Cython module that we can test. To +compile it, we need to configure a ``setup.py`` script for distutils. +Reusing the basic script from the main tutorial:: + + from distutils.core import setup + from distutils.extension import Extension + from Cython.Distutils import build_ext + + setup( + cmdclass = {'build_ext': build_ext}, + ext_modules = [Extension("queue", ["queue.pyx"])] + ) + +We can extend this script to include the necessary setup for building +against the external C library. Assuming it's installed in the normal +places (e.g. under ``/usr/lib`` and ``/usr/include`` on a Unix-like +system), we could simply change the extension setup from :: - ext_modules = [Extension("hello", ["hello.pyx"])] + ext_modules = [Extension("queue", ["queue.pyx"])] to :: ext_modules = [ - Extension("hello", ["hello.pyx"], + Extension("queue", ["queue.pyx"], libraries=["calg"]) ] @@ -167,13 +203,13 @@ flags, such as:: LDFLAGS="-L/usr/local/otherdir/calg/lib" \ python setup.py build_ext -i -Once we have compiled the module for the first time, we can try to -import it:: +Once we have compiled the module for the first time, we can now import +it and instantiate a new Queue:: - PYTHONPATH=. python -c 'import queue.Queue as Q; Q()' + PYTHONPATH=. python -c 'import queue.Queue as Q ; Q()' -However, our class doesn't do much yet so far, so -let's make it more usable. +However, this is all our Queue class can do so far, so let's make it +more usable. Before implementing the public interface of this class, it is good practice to look at what interfaces Python offers, e.g. in its @@ -198,12 +234,12 @@ Here is a simple implementation for the ``append()`` method:: Again, the same error handling considerations as for the ``__cinit__()`` method apply, so that we end up with this -implementation:: +implementation instead:: cdef append(self, int value): if not cqueue.queue_push_tail(self._c_queue, value): - python_exc.PyErr_NoMemory() + cpython.exc.PyErr_NoMemory() Adding an ``extend()`` method should now be straight forward:: @@ -214,7 +250,7 @@ Adding an ``extend()`` method should now be straight forward:: for i in range(count): if not cqueue.queue_push_tail( self._c_queue, values[i]): - python_exc.PyErr_NoMemory() + cpython.exc.PyErr_NoMemory() This becomes handy when reading values from a NumPy array, for example. @@ -230,15 +266,14 @@ which provide read-only and destructive read access respectively:: return cqueue.queue_pop_head(self._c_queue) Simple enough. Now, what happens when the queue is empty? According -to the documentation, the functions return a ``NULL`` pointer, which is -typically not a valid value. Since we are simply casting to and from -ints, we cannot distinguish anymore if the -return value was ``NULL`` because the queue was empty or because the -value stored in the queue was ``0``. However, in Cython code, we -would expect the first case to raise an exception, whereas the second -case should simply return ``0``. To deal with this, we need to -special case this value, and check if the queue really is empty or -not:: +to the documentation, the functions return a ``NULL`` pointer, which +is typically not a valid value. Since we are simply casting to and +from ints, we cannot distinguish anymore if the return value was +``NULL`` because the queue was empty or because the value stored in +the queue was ``0``. However, in Cython code, we would expect the +first case to raise an exception, whereas the second case should +simply return ``0``. To deal with this, we need to special case this +value, and check if the queue really is empty or not:: cdef int peek(self) except? 0: cdef int value = \ @@ -264,30 +299,29 @@ an exception was raised, and if so, propagate the exception. This obviously has a performance penalty. Cython therefore allows you to indicate which value is explicitly returned in the case of an exception, so that the surrounding code only needs to check for an -exception when receiving this special value. All other values will be +exception when receiving this exact value. All other values will be accepted almost without a penalty. -Now that the ``peek()`` method is implemented, the ``pop()`` method is -almost identical. It only calls a different C function:: +Now that the ``peek()`` method is implemented, the ``pop()`` method +also needs adaptation. Since it removes a value from the queue, +however, it is not enough to test if the queue is empty *after* the +removal. Instead, we must test it on entry:: cdef int pop(self) except? 0: - cdef int value = \ - cqueue.queue_pop_head(self._c_queue) - if value == 0: - # this may mean that the queue is empty, or - # that it happens to contain a 0 value - if cqueue.queue_is_empty(self._c_queue): - raise IndexError("Queue is empty") - return value + if cqueue.queue_is_empty(self._c_queue): + raise IndexError("Queue is empty") + return cqueue.queue_pop_head(self._c_queue) Lastly, we can provide the Queue with an emptiness indicator in the -normal Python way:: +normal Python way by defining the ``__bool__()`` special method (note +that Python 2 calls this method ``__nonzero__``, whereas Cython code +can use both):: - def __nonzero__(self): + def __bool__(self): return not cqueue.queue_is_empty(self._c_queue) -Note that this method returns either ``True`` or ``False`` as the -return value of the ``queue_is_empty`` function is declared as a +Note that this method returns either ``True`` or ``False`` as we +declared the return type of the ``queue_is_empty`` function as ``bint``. Now that the implementation is complete, you may want to write some @@ -305,21 +339,17 @@ callable from C code with fast C semantics and without requiring intermediate argument conversion from or to Python types. The following listing shows the complete implementation that uses -``cpdef`` methods where possible. This feature is obviously not -available for the ``extend()`` method, as the method signature is -incompatible with Python argument types. - -:: +``cpdef`` methods where possible:: cimport cqueue - cimport python_exc + cimport cpython.exc cdef class Queue: cdef cqueue.Queue* _c_queue def __cinit__(self): self._c_queue = cqueue.queue_new() if self._c_queue is NULL: - python_exc.PyErr_NoMemory() + cpython.exc.PyErr_NoMemory() def __dealloc__(self): if self._c_queue is not NULL: @@ -328,14 +358,14 @@ incompatible with Python argument types. cpdef append(self, int value): if not cqueue.queue_push_tail(self._c_queue, value): - python_exc.PyErr_NoMemory() + cpython.exc.PyErr_NoMemory() cdef extend(self, int* values, Py_ssize_t count): cdef Py_ssize_t i - for i in range(count): + for i in xrange(count): if not cqueue.queue_push_tail( self._c_queue, values[i]): - python_exc.PyErr_NoMemory() + cpython.exc.PyErr_NoMemory() cpdef int peek(self) except? 0: cdef int value = \ @@ -347,24 +377,37 @@ incompatible with Python argument types. raise IndexError("Queue is empty") return value - cpdef int pop(self) except? 0: - cdef int value = \ - cqueue.queue_pop_head(self._c_queue) - if value == 0: - # this may mean that the queue is empty, - # or that it happens to contain a 0 value - if cqueue.queue_is_empty(self._c_queue): - raise IndexError("Queue is empty") - return value + cdef int pop(self) except? 0: + if cqueue.queue_is_empty(self._c_queue): + raise IndexError("Queue is empty") + return cqueue.queue_pop_head(self._c_queue) - def __nonzero__(self): + def __bool__(self): return not cqueue.queue_is_empty(self._c_queue) -As a quick test with numbers from 0 to 9999 indicates, using this -Queue from Cython code with C ``int`` values is about five times as -fast as using it from Cython code with Python values, almost eight -times faster than using it from Python code in a Python loop, and -still more than twice as fast as using Python's highly optimised -``collections.deque`` type from Cython code with Python integers. +The ``cpdef`` feature is obviously not available for the ``extend()`` +method, as the method signature is incompatible with Python argument +types. However, if wanted, we can rename the C-ish ``extend()`` +method to e.g. ``c_extend()``, and write a new ``extend()`` method +instead that accepts an arbitrary Python iterable:: + + cdef c_extend(self, int* values, Py_ssize_t count): + cdef Py_ssize_t i + for i in range(count): + if not cqueue.queue_push_tail( + self._c_queue, values[i]): + cpython.exc.PyErr_NoMemory() + + cpdef extend(self, values): + for value in values: + self.append(value) + +As a quick test with numbers from 0 to 9999 on the author's machine +indicates, using this Queue from Cython code with C ``int`` values is +about five times as fast as using it from Cython code with Python +values, almost eight times faster than using it from Python code in a +Python loop, and still more than twice as fast as using Python's +highly optimised ``collections.deque`` type from Cython code with +Python integers. .. [CAlg] Simon Howard, C Algorithms library, http://c-algorithms.sourceforge.net/