[[!meta  title="pysawsim.manager: parallelization-framework-agnostic testing"]]
[[!template id=gitrepo repo=sawsim]]

Over the past week I've been working up a [[Python]] framework for my
[[sawsim]] simulator, to allow parallel execution on a variety of
systems.  I think I've got the kinks worked out now, and `sawsim` (or
anything else that can subclass `pysawsim.manager.Job` can now be
transparently executed in parallel using any of

* Python's [threading][] module (the `thread` manager).
* Python's [multiprocessing][] module (the `subproc` manager).
* [[MPI]] (via [mpi4py][], the `mpi` manager).
* [[PBS]] (via [pbs_python][], the `pbs` manager).

Most of the difficulty is in handling systems where not all of these
approaches are viable, (e.g. Python 2.5, so no `multiprocessing`
module, or an SMP host that wants to use `subproc` without installing
mpi4py or pbs_python).  I also ran up against Python's [issue5155][]
when I tried to test the `subproc` manager from within a:

    $ nosetests --with-doctest --doctest-tests --processes $N pysawsim

on a Python 2.6.2 system (it was fixed by 2.6.3).

Note that Python's [GIL restricts threads to a single core][GIL].  This
means that the `thread` manager is basically decorative, but I wrote
it anyway because it forms the basis of several other managers, and
it's the only manager that will definitely work on Python 2.5.

[threading]: http://docs.python.org/library/threading.html
[multiprocessing]: http://docs.python.org/library/multiprocessing.html
[mpi4py]: http://mpi4py.scipy.org/
[pbs_python]: https://subtrac.sara.nl/oss/pbs_python
[issue5155]: http://bugs.python.org/issue5155
[GIL]: http://docs.python.org/faq/library#can-t-we-get-rid-of-the-global-interpreter-lock


[[!tag tags/programming]]
[[!tag tags/python]]
[[!tag tags/sawsim]]