From: Greg Wilson Date: Thu, 22 Aug 2013 01:02:22 +0000 (-0400) Subject: Working on testing lecture for Python X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=c4b87eae74b6c9a7f8e1e999eac2d0fde40cefb9;p=swc-testing-nose.git Working on testing lecture for Python W. Trevor King: This is a IPython Notebook translation of Greg's unit testing content, which he initially developed in 2010 in Software Carpentry's Subverison repository. Unfortunately, the content of that repository was in binary formats (ODF, PNG, etc.), so I can't graft it on here in any useful way. Interested parties should check out: 0a39643 Relocating test material, 2010-08-24 4.0/site/test/unit.html → 4.0/topics/test/unit/test-unit.html baccde4 Script for lecture on unit testing with Nose, 2010-08-18 4.0/site/test/unit.html b37a2f4 Blocking out unit testing lecture, 2010-08-04 ... --- diff --git a/lessons/swc-python/python-5-testing.ipynb b/lessons/swc-python/python-5-testing.ipynb new file mode 100644 index 0000000..fb3d8a2 --- /dev/null +++ b/lessons/swc-python/python-5-testing.ipynb @@ -0,0 +1,1200 @@ +{ + "metadata": { + "name": "" + }, + "nbformat": 3, + "nbformat_minor": 0, + "worksheets": [ + { + "cells": [ + { + "cell_type": "heading", + "level": 1, + "metadata": {}, + "source": [ + "Basic Programming Using Python: Testing" + ] + }, + { + "cell_type": "heading", + "level": 2, + "metadata": {}, + "source": [ + "Objectives" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* FIXME" + ] + }, + { + "cell_type": "heading", + "level": 2, + "metadata": {}, + "source": [ + "Setting Expectations" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We created, found, and fixed over half a dozen bugs\n", + "in our [previous lesson](python-4-files-lists.ipynb).\n", + "How can we be sure that others aren't still lurking in our code?\n", + "It's not an idle worry:\n", + "every year,\n", + "programmers find errors in software that has been in use for years,\n", + "and the number of papers that have been retracted\n", + "because of computational mistakes\n", + "is constantly growing." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The short answer is that it's practically impossible to prove that a program will always do what it's supposed to.\n", + "To see why,\n", + "consider a function that checks whether a character strings contains only 'A', 'C', 'G', and 'T'.\n", + "These four tests clearly aren't sufficient:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "assert is_all_bases('A')\n", + "assert is_all_bases('C')\n", + "assert is_all_bases('G')\n", + "assert is_all_bases('T')\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "because this implementation of `is_all_bases` passes them:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "def is_all_bases(bases):\n", + " return True\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Adding these tests isn't enough:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "assert not is_all_bases('X')\n", + "assert not is_all_bases('Y')\n", + "assert not is_all_bases('Z')\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "because this implementation passes:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "def is_all_bases(bases):\n", + " return bases[0] in 'ACGT'\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can add yet more tests:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "assert is_all_bases('ACGCGA')\n", + "assert not is_all_bases('CGAZ')\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "but no matter how many we have,\n", + "we can always write a function that passes them,\n", + "but does the wrong thing in other cases." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Testing is still worth doing, though:\n", + "it's one of those things that doesn't work in theory,\n", + "but is surprisingly effective in practice.\n", + "If we choose our tests carefully,\n", + "we can demonstrate that our software is as likely to be correct as a mathematical proof\n", + "or a physical experiment." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And ensuring that we have the right answer is only one reason to to software.\n", + "The other is that it speeds up development\n", + "by reducing the amount of re-work we have to do.\n", + "Even small programs can be quite complex,\n", + "and changing one thing can all too easily break something else.\n", + "If we test changes as we make them,\n", + "and re-test things we've already done,\n", + "we can catch errors while the changes are still fresh in our minds,\n", + "which makes fixing them much easier." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It's important to realize,\n", + "though,\n", + "that testing itself doesn't make software better.\n", + "As Steve McConnell once said,\n", + "trying to improve the quality of software by doing more testing\n", + "is like trying to lose weight by weighing yourself more often.\n", + "Testing just tells us what the quality *is*;\n", + "if we want to improve it,\n", + "so that we don't have to throw away a week's worth of analysis because of a missing semi-colon,\n", + "we have to change our programs,\n", + "and change the way we go about writing programs." + ] + }, + { + "cell_type": "heading", + "level": 2, + "metadata": {}, + "source": [ + "Defensive Programming" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The first step is to use [defensive programming](glossary.html#defensive_programming),\n", + "i.e.,\n", + "to put assertions in our programs so that they check their own execution as they run.\n", + "Programs like the Firefox browser are littered with assertions—in fact,\n", + "10-20% of the code they contain\n", + "are there to check that the other 80-90% are working correctly.\n", + "Broadly speaking,\n", + "assertions fall into three categories:\n", + "\n", + "- A [precondition](glossary.html#precondition) is something that must be true\n", + " in order for a piece of code to work correctly.\n", + "- A [postcondition](glossary.html#postcondition) is something that must be true\n", + " at the end of a piece of code if it worked correctly.\n", + "- An [invariant](glossary.html#invariant) is something that is always true\n", + " at a particular point inside a piece of code." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For example,\n", + "suppose we are representing rectangles using a list of four coordinates `[x0, y0, x1, y1]`.\n", + "In order to do some calculations,\n", + "we need to normalize the rectangle so that it is at the origin\n", + "and 1.0 units long on its longest axis.\n", + "This function does that:" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "def normalize_rectangle(rect):\n", + " x0, y0, x1, y1 = rect\n", + " assert x0 < x1, 'Invalid X coordinates'\n", + " assert y0 < y1, 'Invalid Y coordinates'\n", + "\n", + " dx = x1 - x0\n", + " dy = y1 - y0\n", + " if dx > dy:\n", + " scaled = float(dy) / dx\n", + " upper_x, upper_y = 1.0, scaled\n", + " else:\n", + " scaled = float(dx) / dy\n", + " upper_x, upper_y = scaled, 1.0\n", + "\n", + " assert 0 < upper_x <= 1.0, 'Calculated upper X coordinate invalid'\n", + " assert 0 < upper_y <= 1.0, 'Calculated upper Y coordinate invalid'\n", + "\n", + " return [0, 0, upper_x, upper_y]" + ], + "language": "python", + "metadata": {}, + "outputs": [], + "prompt_number": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The first two assertions test that the inputs are valid,\n", + "i.e.,\n", + "that the upper X and Y coordinates are greater than their lower counterparts.\n", + "Notice that the test is greater than,\n", + "not greater than or equal to:\n", + "this tells us (and the computer) that rectangles aren't allowed to have zero width or height.\n", + "The last two assertions check that the upper coordinates of the scaled rectangle are valid:\n", + "neither can be zero\n", + "(because that would mean the rectangle had zero width or height)\n", + "and neither can be greater than 1." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Strictly speaking, these two assertions are redundant:\n", + "if the inputs are correct,\n", + "and our calculation is correct,\n", + "then the last two conditions should always hold.\n", + "But programmers aren't perfect, \n", + "and if there *is* a bug in our calculations,\n", + "we want the program to complain about it as early as possible." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "### *Assertions and Bugs*\n", + "\n", + "\n", + "Another rule that good programmers follow is, \"Bugs become assertions.\"\n", + "Whenever we fix a bug in a program,\n", + "we should add some assertions to the program at that point to catch the bug if it reappears.\n", + "After all,\n", + "if we made the mistake once,\n", + "then we (or someone else) might well make it again.\n", + "Few things are as frustrating as\n", + "having someone delete several carefully-crafted lines of code that fixed a subtle problem\n", + "because they didn't realize what problem those lines were there to fix.\n", + "\n", + "
" + ] + }, + { + "cell_type": "heading", + "level": 2, + "metadata": {}, + "source": [ + "Handling Errors" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Even when programs are careful,\n", + "things sometimes go wrong.\n", + "Some of these errors have external causes,\n", + "like missing or badly-formatted files.\n", + "Others are internal,\n", + "like bugs in code.\n", + "Either way,\n", + "it's actually pretty easy to handle errors in sensible ways." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's start with a look at how programmers used to do error handling.\n", + "Back in the Dark Ages,\n", + "programmers wrote functions to return a [status code](glossary.html#status_code)\n", + "to indicate whether they had run correctly or not.\n", + "This led to programs like this:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "params, status = read_params(param_file)\n", + "if status != OK:\n", + " log.error('Failed to read', param_file)\n", + " sys.exit(ERROR)\n", + "\n", + "grid, status = read_grid(grid_file)\n", + "if status != OK:\n", + " log.error('Failed to read', grid_file)\n", + " sys.exit(ERROR)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The two function calls are all we really want;\n", + "the other six lines to check that files were opened and read properly,\n", + "and to report errors and exit if not." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A lot of code is still written this way,\n", + "but this coding style makes it hard to see the forest for the trees.\n", + "When we're reading a program,\n", + "we want to understand what's supposed to happen when everything works,\n", + "and only then think about what might happen if something goes wrong.\n", + "When the two are interleaved,\n", + "both are harder to understand.\n", + "The net result is that most programmers don't bother to check the status codes their functions return.\n", + "Which means that when errors *do* occur,\n", + "they're even harder to track down." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Luckily, there's a better way.\n", + "Modern languages like Python allow us to use [exceptions](glossary.html#exception) to handle errors.\n", + "More specifically,\n", + "using exceptions allows us to separate the \"normal\" flow of control\n", + "from the \"exceptional\" cases that arise when something goes wrong,\n", + "which makes both easier to understand:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "try:\n", + " params = read_params(param_file)\n", + " grid = read_grid(grid_file)\n", + "except:\n", + " log.error('Failed to read', filename)\n", + " sys.exit(ERROR)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To join the two parts together,\n", + "we use the keywords `try` and `except`.\n", + "These work together like `if` and `else`:\n", + "the statements under the `try` are what should happen if everything works,\n", + "while the statements under `except` are what the program should do if something goes wrong." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We have actually seen exceptions before without knowing it,\n", + "since by default,\n", + "when an exception occurs,\n", + "Python prints it out and halts our program.\n", + "For example,\n", + "trying to open a nonexistent file triggers a type of exception called an `IOError`,\n", + "while an out-of-bounds index to a list triggers an `IndexError`:" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "open('nonexistent-file.txt', 'r')" + ], + "language": "python", + "metadata": {}, + "outputs": [ + { + "ename": "IOError", + "evalue": "[Errno 2] No such file or directory: 'nonexistent-file.txt'", + "output_type": "pyerr", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mIOError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'nonexistent-file.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;31mIOError\u001b[0m: [Errno 2] No such file or directory: 'nonexistent-file.txt'" + ] + } + ], + "prompt_number": 2 + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "values = [0, 1, 2]\n", + "print values[999]" + ], + "language": "python", + "metadata": {}, + "outputs": [ + { + "ename": "IndexError", + "evalue": "list index out of range", + "output_type": "pyerr", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mprint\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m999\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;31mIndexError\u001b[0m: list index out of range" + ] + } + ], + "prompt_number": 3 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can use `try` and `except` to deal with these errors ourselves\n", + "if we don't want the program simply to fall over.\n", + "Here,\n", + "for example,\n", + "we put our attempt to open a nonexistent file inside a `try`,\n", + "and in the `except`, we print a not-very-helpful error message:" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "try:\n", + " reader = open('nonexistent-file.txt', 'r')\n", + "except IOError:\n", + " print 'Whoops!'" + ], + "language": "python", + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "stream": "stdout", + "text": [ + "Whoops!\n" + ] + } + ], + "prompt_number": 6 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When Python executes this code,\n", + "it runs the statement inside the `try`.\n", + "If that works, it skips over the `except` block without running it.\n", + "If an exception occurs inside the `try` block,\n", + "though,\n", + "Python compares the type of the exception to the type specified by the `except`.\n", + "If they match, it executes the code in the `except` block." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`IOError` is Python's way of reporting several kinds of problems\n", + "related to input and output:\n", + "not just files that don't exist,\n", + "but also things like not having permission to read files,\n", + "and so on.\n", + "We can put as many lines of code in a `try` block as we want,\n", + "just as we can put many statements under an `if`.\n", + "We can also handle several different kinds of errors afterward.\n", + "For example,\n", + "here's some code to calculate the entropy at each point in a grid:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "try:\n", + " params = read_params(param_file)\n", + " grid = read_grid(grid_file)\n", + " entropy = lee_entropy(params, grid)\n", + " write_entropy(entropy_file, entropy)\n", + "except IOError:\n", + " log_error_and_exit('IO error')\n", + "except ArithmeticError:\n", + " log_error_and_exit('Arithmetic error')\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Python tries to run the four statements inside the `try` as normal.\n", + "If an error occurs in any of them,\n", + "Python immediately jumps down\n", + "and tries to find an `except` whose type matches the type of the error that occurred.\n", + "If it's an `IOError`,\n", + "Python jumps into the first error handler.\n", + "If it's an `ArithmeticError`,\n", + "Python jumps into the second handler instead.\n", + "It will only execute one of these,\n", + "just as it will only execute one branch\n", + "of a series of `if`/`elif`/`else` statements." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This layout has made the code easier to read,\n", + "but we've lost something important:\n", + "the message printed out by the `IOError` branch doesn't tell us\n", + "which file caused the problem.\n", + "We can do better if we capture and hang on to the object that Python creates\n", + "to record information about the error:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "try:\n", + " params = read_params(param_file)\n", + " grid = read_grid(grid_file)\n", + " entropy = lee_entropy(params, grid)\n", + " write_entropy(entropy_file, entropy)\n", + "except IOError as err:\n", + " log_error_and_exit('Cannot read/write' + err.filename)\n", + "except ArithmeticError as err:\n", + " log_error_and_exit(err.message)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If something goes wrong in the `try`,\n", + "Python creates an exception object,\n", + "fills it with information,\n", + "and assigns it to the variable `err`.\n", + "(There's nothing special about this variable name—we can use anything we want.)\n", + "Exactly what information is recorded depends on what kind of error occurred;\n", + "Python's documentation describes the properties of each type of error in detail,\n", + "but we can always just print the exception object.\n", + "In the case of an I/O error,\n", + "we print out the name of the file that caused the problem.\n", + "And in the case of an arithmetic error,\n", + "printing out the message embedded in the exception object is what Python would have done anyway." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So much for how exceptions work:\n", + "how should they be used?\n", + "Some programmers use `try` and `except` to give their programs default behaviors.\n", + "For example,\n", + "if this code can't read the grid file that the user has asked for,\n", + "it creates a default grid instead:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "try:\n", + " grid = read_grid(grid_file)\n", + "except IOError:\n", + " grid = default_grid()\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Other programmers would explicitly test for the grid file,\n", + "and use `if` and `else` for control flow:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "if file_exists(grid_file):\n", + " grid = read_grid(grid_file)\n", + "else:\n", + " grid = default_grid()\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It's mostly a matter of taste,\n", + "but we prefer the second style.\n", + "As a rule,\n", + "exceptions should only be used to handle exceptional cases.\n", + "If the program knows how to fall back to a default grid,\n", + "that's not an unexpected event.\n", + "Using `if` and `else`\n", + "instead of `try` and `except`\n", + "sends different signals to anyone reading our code,\n", + "even if they do the same thing." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Novices often ask another question about exception handling style as well,\n", + "but before we address it,\n", + "there's something in our example that you might not have noticed.\n", + "Exceptions can actually be thrown a long way:\n", + "they don't have to be handled immediately.\n", + "Take another look at this code:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "try:\n", + " params = read_params(param_file)\n", + " grid = read_grid(grid_file)\n", + " entropy = lee_entropy(params, grid)\n", + " write_entropy(entropy_file, entropy)\n", + "except IOError as err:\n", + " log_error_and_exit('Cannot read/write' + err.filename)\n", + "except ArithmeticError as err:\n", + " log_error_and_exit(err.message)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The four lines in the `try` block are all function calls.\n", + "They might catch and handle exceptions themselves,\n", + "but if an exception occurs in one of them that *isn't* handled internally,\n", + "Python looks in the calling code for a matching `except`.\n", + "If it doesn't find one there,\n", + "it looks in that function's caller,\n", + "and so on.\n", + "If we get all the way back to the main program without finding an exception handler,\n", + "Python's default behavior is to print an error message like the ones we've been seeing all along." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This rule is the origin of the rule \"Throw Low, Catch High.\"\n", + "There are many places in our program where an error might occur.\n", + "There are only a few, though, where errors can sensibly be handled.\n", + "For example,\n", + "a linear algebra library doesn't know whether it's being called directly from the Python interpreter,\n", + "or whether it's being used as a component in a larger program.\n", + "In the latter case,\n", + "the library doesn't know if the program that's calling it is being run from the command line or from a GUI.\n", + "The library therefore shouldn't try to handle or report errors itself,\n", + "because it has no way of knowing what the right way to do this is.\n", + "It should instead just raise an exception,\n", + "and let its caller figure out how best to handle it." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally,\n", + "we can raise exceptions ourselves if we want to.\n", + "In fact,\n", + "we *should* do this,\n", + "since it's the standard way in Python to signal that something has gone wrong.\n", + "Here,\n", + "for example,\n", + "is a function that reads a grid and checks its consistency:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "def read_grid(grid_file):\n", + " '''Read grid, checking consistency.'''\n", + "\n", + " data = read_raw_data(grid_file)\n", + " if not grid_consistent(data):\n", + " raise Exception('Inconsistent grid: ' + grid_file)\n", + " result = normalize_grid(data)\n", + "\n", + " return result\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `raise` statement creates a new exception with a meaningful error message.\n", + "Since `read_grid` itself doesn't contain a `try`/`except` block,\n", + "this exception will always be thrown up and out of the function,\n", + "to be caught and handled by whoever is calling `read_grid`.\n", + "We can define new types of exceptions if we want to.\n", + "And we should,\n", + "so that errors in our code can be distinguished from errors in other people's code.\n", + "However,\n", + "this involves classes and objects,\n", + "which is outside the scope of these lessons." + ] + }, + { + "cell_type": "heading", + "level": 2, + "metadata": {}, + "source": [ + "Unit Testing" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we understand how Python manages error,\n", + "we can return to the subject of testing.\n", + "The biggest obstacle to doing it isn't actually whether or not it's useful,\n", + "but whether or not it's easy to do.\n", + "If it isn't,\n", + "people will always find excuses to do something else.\n", + "It's therefore important to make things as painless as possible.\n", + "In particular, it has to be easy for people to:\n", + "\n", + "- add or change tests,\n", + "- understand the tests that have already been written,\n", + "- run those tests, and\n", + "- understand those tests' results." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Test results must also be reliable to be useful.\n", + "If a testing tool says that code is working when it's not,\n", + "or reports problems when there actually aren't any,\n", + "people will lose faith in it and stop using it." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's start with the simplest kind of testing.\n", + "A [unit test](glossary.html#unit_test) is\n", + "a test that exercises one component, or unit, in a program.\n", + "Every unit test has five parts.\n", + "The first is the [test fixture](glossary.html#test_fixture),\n", + "which is the thing the test is run on:\n", + "the inputs to a function,\n", + "or the data files to be processed." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The second part is the [test action](glossary.html#test_action),\n", + "which is what we do to the fixture.\n", + "Ideally,\n", + "this just involves calling a function,\n", + "but some tests may involve more.\n", + "The third part of every unit test is its [expected result](glossary.html#expected_test_result),\n", + "which is what we expect the piece of code we're testing to do or return.\n", + "If we don't know the expected result,\n", + "we can't tell whether the test passed or failed.\n", + "As we'll see toward the end of this lesson,\n", + "defining fixtures and expected results can be a good way to design software." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The first three parts of the unit test are used over and over again.\n", + "The fourth part is the [actual result](glossary.html#actual_test_result),\n", + "which is what happens when we run the test on a particular day,\n", + "with a particular version of our software.\n", + "The fifth and final part is a [test report](glossary.html#test_report)\n", + "that tells us whether the test passed,\n", + "or whether there's a failure of some kind that needs human attention.\n", + "As with the actual result,\n", + "this could be different each time we run the test." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So much for terminology:\n", + "what does this all look like in practice?\n", + "As an example,\n", + "suppose we're testing a function called `rectangle_area`\n", + "that returns the area of an `[x0, y0, x1, y1]` rectangle.\n", + "We'll start by testing our code directly using `assert`.\n", + "Here,\n", + "we call the function three times with different arguments,\n", + "checking that the right value is returned each time.\n", + "(We import `rectangle_area_1` rather than `rectangle_area`\n", + "because we're going to use several different versions of this function in this lesson,\n", + "and need to give each one a different name.)" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "from rectangle import rectangle_area_1\n", + "\n", + "assert rectangle_area_1([0, 0, 1, 1]) == 1.0\n", + "assert rectangle_area_1([1, 1, 4, 4]) == 9.0\n", + "assert rectangle_area_1([0, 1, 4, 7]) == 24.0" + ], + "language": "python", + "metadata": {}, + "outputs": [ + { + "ename": "AssertionError", + "evalue": "", + "output_type": "pyerr", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mrectangle_area_1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mrectangle_area_1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m9.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mrectangle_area_1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m7\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m24.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;31mAssertionError\u001b[0m: " + ] + } + ], + "prompt_number": 16 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is better than no tests at all,\n", + "but look what happens if we change the order of the tests:" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "assert rectangle_area_1([0, 1, 4, 7]) == 24.0\n", + "assert rectangle_area_1([1, 1, 4, 4]) == 9.0\n", + "assert rectangle_area_1([0, 0, 1, 1]) == 1.0" + ], + "language": "python", + "metadata": {}, + "outputs": [ + { + "ename": "AssertionError", + "evalue": "", + "output_type": "pyerr", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mrectangle_area_1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m7\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m24.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mrectangle_area_1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m9.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mrectangle_area_1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mAssertionError\u001b[0m: " + ] + } + ], + "prompt_number": 17 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Python halts at the first failed assertion,\n", + "so the second and third tests aren't run at all.\n", + "It would be more helpful if we could get data from all of our tests every time they're run,\n", + "since the more information we have,\n", + "the faster we're likely to be able to track down bugs.\n", + "It would also be helpful to have some kind of summary report:\n", + "if our [test suite](glossary.html#test_suite) includes thirty or forty tests\n", + "(as it well might for a complex function or library that's widely used),\n", + "we'd like to know how many passed or failed." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here's a different approach.\n", + "First, let's put each test in a function with a meaningful name:" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "def test_unit_square():\n", + " assert rectangle_area_1([0, 0, 1, 1]) == 1.0\n", + "\n", + "def test_large_square():\n", + " assert rectangle_area_1([1, 1, 4, 4]) == 9.0\n", + "\n", + "def test_actual_rectangle():\n", + " assert rectangle_area_1([0, 1, 4, 7]) == 24.0" + ], + "language": "python", + "metadata": {}, + "outputs": [], + "prompt_number": 24 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next,\n", + "we'll import a library called `ears`\n", + "and ask it to run our tests for us:" + ] + }, + { + "cell_type": "code", + "collapsed": false, + "input": [ + "import ears\n", + "ears.run()" + ], + "language": "python", + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "stream": "stdout", + "text": [ + "..f\n", + "2 pass, 1 fail, 0 error\n", + "----------------------------------------\n", + "fail: test_actual_rectangle\n", + "Traceback (most recent call last):\n", + " File \"ears.py\", line 43, in run\n", + " test()\n", + " File \"\", line 8, in test_actual_rectangle\n", + " assert rectangle_area_1([0, 1, 4, 7]) == 24.0\n", + "AssertionError\n", + "\n" + ] + } + ], + "prompt_number": 25 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`ears.run` looks in the calling program\n", + "for functions whose names start with the letters `'test_'`\n", + "and runs each one exactly once.\n", + "If the function complete without an assertion being triggered,\n", + "we count the test as a success.\n", + "If an assertion fails,\n", + "we count the test as a failure,\n", + "and if any other exception occurs,\n", + "we count it as an error\n", + "(i.e., we assume that the test itself is broken)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`ears` belongs to a family of tools called [xUnit testing library](glossary.html#xUnit).\n", + "The name \"xUnit\" comes from the fact that\n", + "many of them are imitations of a Java testing library called JUnit.\n", + "The [Wikipedia page](http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks) on the subject\n", + "lists dozens of similar frameworks in almost as many languages,\n", + "all of which have a similar structure:\n", + "each test is a single function that follows some naming convention\n", + "(e.g., starts with `'test_'`),\n", + "and the framework runs them in some order\n", + "and reports how many passed, failed, or were broken." + ] + }, + { + "cell_type": "heading", + "level": 2, + "metadata": {}, + "source": [ + "Test-Driven Development" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Of course,\n", + "these libraries can't think of test cases for us.\n", + "We still have to decide what to test and how many tests to run.\n", + "Our best guide here is economics:\n", + "we want the tests that are most likely to give us useful information\n", + "that we don't already have.\n", + "For example,\n", + "if `rectangle_area([0, 0, 1, 1])` works,\n", + "there's probably not much point testing `rectangle_area([0, 0, 2, 2])`,\n", + "since it's hard to think of a bug that would show up in one case but not in the other." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We should therefore try to choose tests that are as different from each other as possible,\n", + "so that we force the code we're testing to execute in all the different ways it can.\n", + "Another way of thinking about this is that we should try to find [boundary cases](glossary.html#boundary_case).\n", + "If a function works for zero,\n", + "one,\n", + "and a million values,\n", + "it will probably work for eighteen values." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using boundary values as tests has another advantage:\n", + "it can help us design our software.\n", + "To see how,\n", + "consider this test case for our rectangle area function:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "def test_inverted_rectangle():\n", + " assert rectangle_area([1, 5, 5, 2]) == -12.0\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Is that test correct?\n", + "I.e.,\n", + "are rectangles with `x1