From 23701851abc088f7d6df6d7413cdc638cbffda39 Mon Sep 17 00:00:00 2001 From: "W. Trevor King" Date: Wed, 27 Oct 2010 15:59:04 -0400 Subject: [PATCH] Add pysawsim-manager post. --- posts/pysawsim-manager.mdwn | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 posts/pysawsim-manager.mdwn diff --git a/posts/pysawsim-manager.mdwn b/posts/pysawsim-manager.mdwn new file mode 100644 index 0000000..e8ce2cd --- /dev/null +++ b/posts/pysawsim-manager.mdwn @@ -0,0 +1,40 @@ +[[!meta title="pysawsim.manager: parallelization-framework-agnostic testing"]] +[[!template id=gitrepo repo=sawsim]] + +Over the past week I've been working up a [[Python]] framework for my +[[sawsim]] simulator, to allow parallel execution on a variety of +systems. I think I've got the kinks worked out now, and `sawsim` runs +(or anything else that can subclass `pysawsim.manager.Job` can now be +transparently executed in parallel using any of + +* Python's [threading][] module (the `thread` manager). +* Python's [multiprocessing][] module (the `subproc` manager). +* [[MPI]] (via [mpi4py][], the `mpi` manager). +* [[PBS]] (via [pbs_python][], the `pbs` manager). + +Most of the difficulty is in handling systems where not all of these +approaches are viable, (e.g. Python 2.5, so no `multiprocessing` +module, or an SMP host that wants to use `subproc` without installing +mpi4py or pbs_python). I also ran up against Python's [issue5155][] +when I tried to test the `subproc` manager from within a: + + $ nosetests --with-doctest --doctest-tests --processes $N pysawsim + +on a Python 2.6.2 system (it was fixed by 2.6.3). + +Note that Python's [GIL restricts threads to a single core][GIL]. This +means that the `thread` manager is basically decorative, but I wrote +it anyway because it forms the basis of several other managers, and +it's the only manager that will definitely work on Python 2.5. + +[threading]: http://docs.python.org/library/threading.html +[multiprocessing]: http://docs.python.org/library/multiprocessing.html +[mpi4py]: http://mpi4py.scipy.org/ +[pbs_python]: https://subtrac.sara.nl/oss/pbs_python +[issue5155]: http://bugs.python.org/issue5155 +[GIL]: http://docs.python.org/faq/library#can-t-we-get-rid-of-the-global-interpreter-lock + + +[[!tag tags/programming]] +[[!tag tags/python]] +[[!tag tags/sawsim]] -- 2.26.2