Theories

I've just released a slightly updated version of my NUnit extension for data-driven unit testing.

There's been a lot of discussion on the NUnit developer list recently regarding Theories - something new in JUnit and xUnit.Net, and it's taken a while to discover why they're so powerful (they're superficially very similar to data-driven unit tests, and a lot of the differences are semantics).

First, there's some good background on theories written by David Saff:
http://shareandenjoy.saff.net/tdd-specifications.pdf
http://shareandenjoy.saff.net/2007/04/popper-and-junitfactory.html
http://dspace.mit.edu/bitstream/1721.1/40090/1/MIT-CSAIL-TR-2008-002.pdf

Theories on first glance look like a data-driven unit test, but I think that the most important difference is, is that:

Theories are, in theory (excuse the pun), supposed to pass for ANY POSSIBLE parameters, whereas data-driven tests only express the behaviour examples that the developer has provided (they are nothing new in unit testing - just a way for a developer to more clearly group parameters together, or get the parameterized data from an external data source without recompiling tests).

Theories are a generalized statement of how the program should run, whereas in TDDing, a very explicit statement of intent is made, which can be made to pass by coding that specific case in the implementation, and then the program is made to work by triangulization - expressing the generalization by giving more inputs. However, the theory literature points out that as we haven't passed in too many data points we can't be sure whether we've actually expressed what we meant.

Theories, by forcing us to write our tests such that they take any inputs, are much more powerful a statement, and allow for the possible inputs to be explored with external tools.

As an aside, one question I posted to the NUnit developer list regarding theories: "One thing that comes to mind, is that theories are written such that all possible inputs should pass. Apart from using a tool such as agitator, is there a way to test that the tests are written in a general way (I mean, if you had a theory that took parameters, but it totally ignored those parameters and worked as a vanilla unit test - i.e. created its own input), then it's not really a valid theory - is
there a way to detect these cases? Probably not, but I was just idly wondering." Answers on a postcard to... well, I'd prefer a reply comment ;-)

Labels: , ,