=================
Apart from writing fast code, one of the main use cases of Cython is
-to call external C libraries from Python code. As seen for the C
-string decoding functions above, it is actually trivial to call C
+to call external C libraries from Python code. As Cython code
+compiles down to C code itself, it is actually trivial to call C
functions directly in the code. The following describes what needs to
be done to use an external C library in Cython code.
The C API of the queue implementation, which is defined in the header
file ``libcalg/queue.h``, essentially looks like this::
+ /* file: queue.h */
+
typedef struct _Queue Queue;
typedef void *QueueValue;
To get started, the first step is to redefine the C API in a ``.pxd``
file, say, ``cqueue.pxd``::
+ # file: cqueue.pxd
+
cdef extern from "libcalg/queue.h":
ctypedef struct Queue:
pass
bint queue_is_empty(Queue* queue)
Note how these declarations are almost identical to the header file
-declarations, so you can often just copy them over. One exception is
-the last line. The return value of the ``queue_is_empty`` method is
-actually a C boolean value, i.e. it is either zero or non-zero,
-indicating if the queue is empty or not. This is best expressed by
-Cython's ``bint`` type, which is a normal ``int`` type when used in C
-but maps to Python's boolean values ``True`` and ``False`` when
-converted to a Python object. Another difference is the first
-line. ``Queue`` is in this case used as an *opaque handle*; only the
-library that is called know what is actually inside. Since no Cython code
-needs to know the contents of the struct, we do not need to declare its contents,
-so we simply provide an empty definition (as we do not want to declare
-the ``_Queue`` type which is referenced in the C header) [#]_.
+declarations, so you can often just copy them over. One noteworthy
+difference is the first line. ``Queue`` is in this case used as an
+*opaque handle*; only the library that is called knows what is really
+inside. Since no Cython code needs to know the contents of the
+struct, we do not need to declare its contents, so we simply provide
+an empty definition (as we do not want to declare the ``_Queue`` type
+which is referenced in the C header) [#]_.
.. [#] There's a subtle difference between ``cdef struct Queue: pass``
- and ``ctypedef struct Queue: pass``. The former declares a type
- which is referenced in C code as ``struct Queue``, while the
- latter is referenced in C as ``Queue``. This is a C language
- quirk that Cython is not able to hide. Most modern C libraries
- use the ``ctypedef`` kind of struct.
+ and ``ctypedef struct Queue: pass``. The former declares a
+ type which is referenced in C code as ``struct Queue``, while
+ the latter is referenced in C as ``Queue``. This is a C
+ language quirk that Cython is not able to hide. Most modern C
+ libraries use the ``ctypedef`` kind of struct.
+
+Another exception is the last line. The integer return value of the
+``queue_is_empty`` method is actually a C boolean value, i.e. it is
+either zero or non-zero, indicating if the queue is empty or not.
+This is best expressed by Cython's ``bint`` type, which is a normal
+``int`` type when used in C but maps to Python's boolean values
+``True`` and ``False`` when converted to a Python object.
Next, we need to design the Queue class that should wrap the C queue.
+It will live in a file called ``queue.pyx``. [#]_
+
+.. [#] Note that the name of the ``.pyx`` file must be different from
+ the ``cqueue.pxd`` file with declarations from the C library,
+ as both do not describe the same code. A ``.pxd`` file next to
+ a ``.pyx`` file with the same name defines exported
+ declarations for code in the ``.pyx`` file.
+
Here is a first start for the Queue class::
+ # file: queue.pyx
+
cimport cqueue
- cimport python_exc
cdef class Queue:
cdef cqueue.Queue _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
-Note that it says ``__cinit__`` rather than ``__init__``. While
+Note that it says ``__cinit__`` rather than ``__init__``. While
``__init__`` is available as well, it is not guaranteed to be run (for
-instance, one could create a subclass and forget to call the ancestor
-constructor). Because not initializing C pointers often leads to
-crashing the Python interpreter without leaving as much as a stack
-trace, Cython provides ``__cinit__`` which is *always* called on
-construction. However, as ``__cinit__`` is called during object
-construction, ``self`` is not fully
-constructed yet, and one must avoid doing anything with ``self`` but
-assigning to ``cdef`` fields.
+instance, one could create a subclass and forget to call the
+ancestor's constructor). Because not initializing C pointers often
+leads to crashing the Python interpreter without leaving as much as a
+stack trace, Cython provides ``__cinit__`` which is *always* called on
+construction. However, as ``__cinit__`` is called during object
+construction, ``self`` is not fully constructed yet, and one must
+avoid doing anything with ``self`` but assigning to ``cdef`` fields.
Note also that the above method takes no parameters, although subtypes
may want to accept some. Although it is guaranteed to get called, the
no-arguments ``__cinit__()`` method is a special case here as it does
-not prevent subclasses from adding parameters as they see fit. If
-parameters are added they must match those of any declared ``__init__``
-method.
+not prevent subclasses from adding parameters as they see fit. If
+parameters are added they must match those of any declared
+``__init__`` method.
Before we continue implementing the other methods, it is important to
understand that the above implementation is not safe. In case
that case, it will return ``NULL``, whereas it would normally return a
pointer to the new queue.
-The normal way to get out of this is to raise an exception, but
-allocating a new exception instance may actually fail when we are
-running out of memory. Luckily, CPython provides a function
-``PyErr_NoMemory()`` that raises the right exception for us. We can
-thus change the init function as follows::
+The normal Python way to get out of this is to raise an exception, but
+in this specific case, allocating a new exception instance may
+actually fail because we are running out of memory. Luckily, CPython
+provides a function ``PyErr_NoMemory()`` that safely raises the right
+exception for us. We can thus change the init function as follows::
+ cimport cpython.exc # standard cimport from CPython's C-API
+ cimport cqueue
+
+ cdef class Queue:
+ cdef cqueue.Queue _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
if self._c_queue is NULL:
- python_exc.PyErr_NoMemory()
-
-The next thing to do is to clean up when the Queue is no longer used.
-To this end, CPython provides a callback that Cython makes available
-as a special method ``__dealloc__()``. In our case, all we have to do
-is to free the Queue, but only if we succeeded in initialising it in
+ cpython.exc.PyErr_NoMemory()
+
+The ``cpython`` package contains pre-defined ``.pxd`` files that ship
+with Cython. If you need any CPython C-API functions, you can cimport
+them from this package. See Cython's ``Cython/Includes/`` source
+package for a complete list of ``.pxd`` files, including parts of the
+standard C library.
+
+The next thing to do is to clean up when the Queue instance is no
+longer used (i.e. all references to it have been deleted). To this
+end, CPython provides a callback that Cython makes available as a
+special method ``__dealloc__()``. In our case, all we have to do is
+to free the C Queue, but only if we succeeded in initialising it in
the init method::
def __dealloc__(self):
if self._c_queue is not NULL:
cqueue.queue_free(self._c_queue)
-At this point, we have a compilable Cython module that we can test.
-To compile it, we need to configure a ``setup.py`` script for
-distutils. Based on the example presented earlier on, we can extend
-the script to include the necessary setup for building against the
-external C library. Assuming it's installed in the normal places
-(e.g. under ``/usr/lib`` and ``/usr/include`` on a Unix-like system),
-we could simply change the extension setup from
+At this point, we have a working Cython module that we can test. To
+compile it, we need to configure a ``setup.py`` script for distutils.
+Reusing the basic script from the main tutorial::
+
+ from distutils.core import setup
+ from distutils.extension import Extension
+ from Cython.Distutils import build_ext
+
+ setup(
+ cmdclass = {'build_ext': build_ext},
+ ext_modules = [Extension("queue", ["queue.pyx"])]
+ )
+
+We can extend this script to include the necessary setup for building
+against the external C library. Assuming it's installed in the normal
+places (e.g. under ``/usr/lib`` and ``/usr/include`` on a Unix-like
+system), we could simply change the extension setup from
::
- ext_modules = [Extension("hello", ["hello.pyx"])]
+ ext_modules = [Extension("queue", ["queue.pyx"])]
to
::
ext_modules = [
- Extension("hello", ["hello.pyx"],
+ Extension("queue", ["queue.pyx"],
libraries=["calg"])
]
LDFLAGS="-L/usr/local/otherdir/calg/lib" \
python setup.py build_ext -i
-Once we have compiled the module for the first time, we can try to
-import it::
+Once we have compiled the module for the first time, we can now import
+it and instantiate a new Queue::
- PYTHONPATH=. python -c 'import queue.Queue as Q; Q()'
+ PYTHONPATH=. python -c 'import queue.Queue as Q ; Q()'
-However, our class doesn't do much yet so far, so
-let's make it more usable.
+However, this is all our Queue class can do so far, so let's make it
+more usable.
Before implementing the public interface of this class, it is good
practice to look at what interfaces Python offers, e.g. in its
Again, the same error handling considerations as for the
``__cinit__()`` method apply, so that we end up with this
-implementation::
+implementation instead::
cdef append(self, int value):
if not cqueue.queue_push_tail(self._c_queue,
<void*>value):
- python_exc.PyErr_NoMemory()
+ cpython.exc.PyErr_NoMemory()
Adding an ``extend()`` method should now be straight forward::
for i in range(count):
if not cqueue.queue_push_tail(
self._c_queue, <void*>values[i]):
- python_exc.PyErr_NoMemory()
+ cpython.exc.PyErr_NoMemory()
This becomes handy when reading values from a NumPy array, for
example.
return <int>cqueue.queue_pop_head(self._c_queue)
Simple enough. Now, what happens when the queue is empty? According
-to the documentation, the functions return a ``NULL`` pointer, which is
-typically not a valid value. Since we are simply casting to and from
-ints, we cannot distinguish anymore if the
-return value was ``NULL`` because the queue was empty or because the
-value stored in the queue was ``0``. However, in Cython code, we
-would expect the first case to raise an exception, whereas the second
-case should simply return ``0``. To deal with this, we need to
-special case this value, and check if the queue really is empty or
-not::
+to the documentation, the functions return a ``NULL`` pointer, which
+is typically not a valid value. Since we are simply casting to and
+from ints, we cannot distinguish anymore if the return value was
+``NULL`` because the queue was empty or because the value stored in
+the queue was ``0``. However, in Cython code, we would expect the
+first case to raise an exception, whereas the second case should
+simply return ``0``. To deal with this, we need to special case this
+value, and check if the queue really is empty or not::
cdef int peek(self) except? 0:
cdef int value = \
obviously has a performance penalty. Cython therefore allows you to
indicate which value is explicitly returned in the case of an
exception, so that the surrounding code only needs to check for an
-exception when receiving this special value. All other values will be
+exception when receiving this exact value. All other values will be
accepted almost without a penalty.
-Now that the ``peek()`` method is implemented, the ``pop()`` method is
-almost identical. It only calls a different C function::
+Now that the ``peek()`` method is implemented, the ``pop()`` method
+also needs adaptation. Since it removes a value from the queue,
+however, it is not enough to test if the queue is empty *after* the
+removal. Instead, we must test it on entry::
cdef int pop(self) except? 0:
- cdef int value = \
- <int>cqueue.queue_pop_head(self._c_queue)
- if value == 0:
- # this may mean that the queue is empty, or
- # that it happens to contain a 0 value
- if cqueue.queue_is_empty(self._c_queue):
- raise IndexError("Queue is empty")
- return value
+ if cqueue.queue_is_empty(self._c_queue):
+ raise IndexError("Queue is empty")
+ return <int>cqueue.queue_pop_head(self._c_queue)
Lastly, we can provide the Queue with an emptiness indicator in the
-normal Python way::
+normal Python way by defining the ``__bool__()`` special method (note
+that Python 2 calls this method ``__nonzero__``, whereas Cython code
+can use both)::
- def __nonzero__(self):
+ def __bool__(self):
return not cqueue.queue_is_empty(self._c_queue)
-Note that this method returns either ``True`` or ``False`` as the
-return value of the ``queue_is_empty`` function is declared as a
+Note that this method returns either ``True`` or ``False`` as we
+declared the return type of the ``queue_is_empty`` function as
``bint``.
Now that the implementation is complete, you may want to write some
intermediate argument conversion from or to Python types.
The following listing shows the complete implementation that uses
-``cpdef`` methods where possible. This feature is obviously not
-available for the ``extend()`` method, as the method signature is
-incompatible with Python argument types.
-
-::
+``cpdef`` methods where possible::
cimport cqueue
- cimport python_exc
+ cimport cpython.exc
cdef class Queue:
cdef cqueue.Queue* _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
if self._c_queue is NULL:
- python_exc.PyErr_NoMemory()
+ cpython.exc.PyErr_NoMemory()
def __dealloc__(self):
if self._c_queue is not NULL:
cpdef append(self, int value):
if not cqueue.queue_push_tail(self._c_queue,
<void*>value):
- python_exc.PyErr_NoMemory()
+ cpython.exc.PyErr_NoMemory()
cdef extend(self, int* values, Py_ssize_t count):
cdef Py_ssize_t i
- for i in range(count):
+ for i in xrange(count):
if not cqueue.queue_push_tail(
self._c_queue, <void*>values[i]):
- python_exc.PyErr_NoMemory()
+ cpython.exc.PyErr_NoMemory()
cpdef int peek(self) except? 0:
cdef int value = \
raise IndexError("Queue is empty")
return value
- cpdef int pop(self) except? 0:
- cdef int value = \
- <int>cqueue.queue_pop_head(self._c_queue)
- if value == 0:
- # this may mean that the queue is empty,
- # or that it happens to contain a 0 value
- if cqueue.queue_is_empty(self._c_queue):
- raise IndexError("Queue is empty")
- return value
+ cdef int pop(self) except? 0:
+ if cqueue.queue_is_empty(self._c_queue):
+ raise IndexError("Queue is empty")
+ return <int>cqueue.queue_pop_head(self._c_queue)
- def __nonzero__(self):
+ def __bool__(self):
return not cqueue.queue_is_empty(self._c_queue)
-As a quick test with numbers from 0 to 9999 indicates, using this
-Queue from Cython code with C ``int`` values is about five times as
-fast as using it from Cython code with Python values, almost eight
-times faster than using it from Python code in a Python loop, and
-still more than twice as fast as using Python's highly optimised
-``collections.deque`` type from Cython code with Python integers.
+The ``cpdef`` feature is obviously not available for the ``extend()``
+method, as the method signature is incompatible with Python argument
+types. However, if wanted, we can rename the C-ish ``extend()``
+method to e.g. ``c_extend()``, and write a new ``extend()`` method
+instead that accepts an arbitrary Python iterable::
+
+ cdef c_extend(self, int* values, Py_ssize_t count):
+ cdef Py_ssize_t i
+ for i in range(count):
+ if not cqueue.queue_push_tail(
+ self._c_queue, <void*>values[i]):
+ cpython.exc.PyErr_NoMemory()
+
+ cpdef extend(self, values):
+ for value in values:
+ self.append(value)
+
+As a quick test with numbers from 0 to 9999 on the author's machine
+indicates, using this Queue from Cython code with C ``int`` values is
+about five times as fast as using it from Cython code with Python
+values, almost eight times faster than using it from Python code in a
+Python loop, and still more than twice as fast as using Python's
+highly optimised ``collections.deque`` type from Cython code with
+Python integers.
.. [CAlg] Simon Howard, C Algorithms library, http://c-algorithms.sourceforge.net/
projects / cython.git / commitdiff