From: W. Trevor King Date: Sat, 21 Jul 2012 20:07:56 +0000 (-0400) Subject: Move 'Reading IGOR...' post to 'igor' and link to igor.git. X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=9bb8527582229fe6b7ff6fbac9e842c3c10533f1;p=blog.git Move 'Reading IGOR...' post to 'igor' and link to igor.git. --- diff --git a/posts/Reading_IGOR_binary_waves_from_Python.mdwn b/posts/Reading_IGOR_binary_waves_from_Python.mdwn deleted file mode 100644 index bd2fb69..0000000 --- a/posts/Reading_IGOR_binary_waves_from_Python.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -[[!meta title="Reading IGOR binary waves from Python"]] -[[!meta date="2010-06-04 18:48:46"]] - -I finally got around to translating some IBW readers from C to Python. -And so, I present (drumroll :) [igorbinarywave.py][] Python because -it's easy to drop it into my other Python projects (in this case, -[[Hooke]]). It's also easy to get a hold of all that useful metadata -in a hurry. No writing ability yet, but I don't know why you'd want -to move data that direction anyway ;). - -Thanks to the folks at [WaveMetrics][] for publishing some good -[documentation][], even if it's in a goofy format. - -Note that if you're designing a system, [[HDF5]] is almost certainly a -better choice for your data file format than IBW. This module exists -for those of you who's data is already stuck in IBW. - -[igorbinarywave.py]: http://git.tremily.us/?p=hooke.git;a=blob;f=hooke/util/igorbinarywave.py;hb=HEAD -[WaveMetrics]: http://www.wavemetrics.com/ -[documentation]: ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/TN003.zip - -[[!tag tags/programming]] diff --git a/posts/igor.mdwn b/posts/igor.mdwn new file mode 100644 index 0000000..25cad56 --- /dev/null +++ b/posts/igor.mdwn @@ -0,0 +1,142 @@ +[[!meta title="Reading IGOR files from Python"]] + +This is the home page for the `igor` package, [[Python]] modules for +reading files written by [WaveMetrics][] IGOR Pro. Note that if +you're designing a system, [[HDF5]] is almost certainly a better +choice for your data file format than IBW or PXP. This package exists +for those of you who's data is already stuck in an IGOR format. + +History +------- + +When I joined Prof. Yang's lab, there was a good deal of data analysis +code written in IGOR, and a bunch of old data saved in IGOR binary +wave (IBW) and packed experiment (PXP) files. I don't use MS Windows, +so I don't run IGOR, but I still needed a way to get at the data. +Luckily, the [WaveMetrics][] folks publish [some useful notes][TN] +which explain the fundamentals of these two file formats ([TN003][] +for IBW and [PTN003][] for PXP). The file formats are in a goofy +format, but [strings][] pulls out enough meat to figure out what's +going on. + +For a while I used a IBW → ASCII reader that I coded up in [[C]], but +when I joined the [[Hooke]] project during the winter of 2009–2010, I +translated the reader into [[Python]] to support the drivers for data +from Asylum Research's [MFP-*][MFP-1D] and related microscopes. This +scratched my itch for a few years. + +Fast forward to 2012, and for the first time I needed to extract data +from a PXP file. Since my Python code only supported IBW's, I +searched around and found [igor.py][] by Paul Kienzle Merlijn van +Deen. They had a PXP reader, but no reader for stand-alone IBW files. +I decided to merge the two projects, so I split my reader out of the +Hooke repository and hacked up the [[Git]] repository referenced +above. Now it's easy to get a hold of all that useful metadata in a +hurry. No writing ability yet, but I don't know why you'd want to +move data that direction anyway ;). + +Parsing dynamic structures with Python +-------------------------------------- + +The IGOR file formats rely on lots of shenanigans with C `struct`s. +To meld all the structures together in a natural way, I've extended +Python's standard [struct][] library to support arbitrary nesting and +dynamic fields. Take a look at [igor.struct][struct.py] for some +examples. This framework makes it easy to load data from structures +like: + + struct vector { + unsigned int length; + short data[length]; + }; + +With the standard `struct` module, you'd read this using the +functional approach: + + >>> import struct + >>> buffer = b'\x00\x00\x00\x02\x01\x02\x03\x04' + >>> length_struct = struct.Struct('>I') + >>> length = length_struct.unpack_from(buffer)[0] + >>> data = struct.unpack_from('>' + 'h'*length, buffer, length_struct.size) + >>> print(data) + (258, 772) + +This obviously works, but keeping track of the offsets, byte ordering, +etc. can be tedious. My `igor.struct` package allows you to use a +more object oriented approach: + + >>> from pprint import pprint + >>> from igor.struct import Field, DynamicField, DynamicStructure + >>> class DynamicLengthField (DynamicField): + ... def pre_pack(self, parents, data): + ... "Set the 'length' value to match the data before packing" + ... vector_structure = parents[-1] + ... vector_data = self._get_structure_data( + ... parents, data, vector_structure) + ... length = len(vector_data['data']) + ... vector_data['length'] = length + ... data_field = vector_structure.get_field('data') + ... data_field.count = length + ... data_field.setup() + ... def post_unpack(self, parents, data): + ... "Adjust the expected data count to match the 'length' value" + ... vector_structure = parents[-1] + ... vector_data = self._get_structure_data( + ... parents, data, vector_structure) + ... length = vector_data['length'] + ... data_field = vector_structure.get_field('data') + ... data_field.count = length + ... data_field.setup() + >>> dynamic_length_vector = DynamicStructure('vector', + ... fields=[ + ... DynamicLengthField('I', 'length'), + ... Field('h', 'data', count=0, array=True), + ... ], + ... byte_order='>') + >>> vector = dynamic_length_vector.unpack(buffer) + >>> pprint(vector) + {'data': array([258, 772]), 'length': 2} + +While this is overkill for such a simple example, it scales much more +cleanly than an approach using the standard `struct` module. The main +benefit is that you can use `Structure` instances as format specifiers +for `Field` instances. This means that you could specify a C +structure like: + + struct vectors { + unsigned int length; + struct vector data[length]; + }; + +With: + + >>> dynamic_length_vectors = DynamicStructure('vectors', + ... fields=[ + ... DynamicLengthField('I', 'length'), + ... Field(dynamic_length_vector, 'data', count=0, array=True), + ... ], + ... byte_order='>') + +The C code your mimicking probably only uses a handful of dynamic +approaches. Once you've written classes to handle each of them, it is +easy to translate arbitrarily complex nested C structures into Python +representations. + +The pre-pack and post-unpack hooks also give you a convenient place to +translate between some C struct's funky format and Python's native +types. You take care off all that when you define the structure, and +then any part of your software that uses the structure gets the native +version automatically. + + +[WaveMetrics]: http://www.wavemetrics.com/ +[TN]: ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/ +[TN003]: ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/TN003.zip +[PTN003]: ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/PTN003.zip +[strings]: http://www.gnu.org/software/binutils/ +[MFP-1D]: http://www.asylumresearch.com/Products/Mfp1D/Mfp1D.shtml +[igor.py]: http://pypi.python.org/pypi/igor.py +[struct]: http://docs.python.org/library/struct.html +[struct.py]: http://git.tremily.us/?p=igor.git;a=blob;f=igor/struct.py;hb=HEAD + +[[!tag tags/programming]]