# When should we test? Short answers: - **ALWAYS!** - **EARLY!** - **OFTEN!** Long answers: * Definitely before you do something important with your software (e.g. publishing data generated by your program, launching a satellite that depends on your software, …). * Before and after adding something new, to avoid accidental breakage. * To help remember ([TDD][]: define) what your code actually does. # Who should test? * Write tests for the stuff you code, to convince your collaborators that it works. * Write tests for the stuff others code, to convince yourself that it works (and will continue to work). Professionals often test their code, and take pride in test coverage, the percent of their functions that they feel confident are comprehensively tested. # How are tests written? The type of tests that are written is determined by the testing framework you adopt. Don't worry, there are a lot of choices. ## Types of Tests **Exceptions:** Exceptions can be thought of as type of runtime test. They alert the user to exceptional behavior in the code. Often, exceptions are related to functions that depend on input that is unknown at compile time. Checks that occur within the code to handle exceptional behavior that results from this type of input are called Exceptions. **Unit Tests:** Unit tests are a type of test which test the fundamental units of a program's functionality. Often, this is on the class or function level of detail. However what defines a *code unit* is not formally defined. To test functions and classes, the interfaces (API) - rather than the implementation - should be tested. Treating the implementation as a black box, we can probe the expected behavior with boundary cases for the inputs. **System Tests:** System level tests are intended to test the code as a whole. As opposed to unit tests, system tests ask for the behavior as a whole. This sort of testing involves comparison with other validated codes, analytical solutions, etc. **Regression Tests:** A regression test ensures that new code does change anything. If you change the default answer, for example, or add a new question, you'll need to make sure that missing entries are still found and fixed. **Integration Tests:** Integration tests query the ability of the code to integrate well with the system configuration and third party libraries and modules. This type of test is essential for codes that depend on libraries which might be updated independently of your code or when your code might be used by a number of users who may have various versions of libraries. **Test Suites:** Putting a series of unit tests into a collection of modules creates, a test suite. Typically the suite as a whole is executed (rather than each test individually) when verifying that the code base still functions after changes have been made. # Elements of a Test **Behavior:** The behavior you want to test. For example, you might want to test the fun() function. **Expected Result:** This might be a single number, a range of numbers, a new fully defined object, a system state, an exception, etc. When we run the fun() function, we expect to generate some fun. If we don't generate any fun, the fun() function should fail its test. Alternatively, if it does create some fun, the fun() function should pass this test. The the expected result should known *a priori*. For numerical functions, this is result is ideally analytically determined even if the function being tested isn't. **Assertions:** Require that some conditional be true. If the conditional is false, the test fails. **Fixtures:** Sometimes you have to do some legwork to create the objects that are necessary to run one or many tests. These objects are called fixtures as they are not really part of the test themselves but rather involve getting the computer into the appropriate state. For example, since fun varies a lot between people, the fun() function is a method of the Person class. In order to check the fun function, then, we need to create an appropriate Person object on which to run fun(). **Setup and teardown:** Creating fixtures is often done in a call to a setup function. Deleting them and other cleanup is done in a teardown function. **The Big Picture:** Putting all this together, the testing algorithm is often: ```python setup() test() teardown() ``` But, sometimes it's the case that your tests change the fixtures. If so, it's better for the setup() and teardown() functions to occur on either side of each test. In that case, the testing algorithm should be: ```python setup() test1() teardown() setup() test2() teardown() setup() test3() teardown() ``` # Test-Driven Development [Test-driven development][TDD] (TDD) is a philosophy whereby the developer creates code by **writing the tests first**. That is to say you write the tests *before* writing the associated code! This is an iterative process whereby you write a test then write the minimum amount code to make the test pass. If a new feature is needed, another test is written and the code is expanded to meet this new use case. This continues until the code does what is needed. TDD operates on the YAGNI principle (You Ain't Gonna Need It). People who diligently follow TDD swear by its effectiveness. This development style was put forth most strongly by [Kent Beck in 2002][KB]. [TDD]: http://en.wikipedia.org/wiki/Test-driven_development [KB]: http://www.amazon.com/Test-Driven-Development-By-Example/dp/0321146530