Sample Theory Implementation as NUnit Extension.

There's been lots of comments bouncing around on the NUnit mailing list about what exactly constitutes a Theory, and what the desired features are, so I've created an NUnit extension with a sample Theory implementation - you can get it, Maslina version 1.0.0.0, from www.taumuon.co.uk/rakija

xUnit.Net implements theories but does not have any in-built Assumption mechanism (you can effectively filter out bad data, which is the same as a filtering assumption). JUnit 4.4, I think, only filters out data - it doesn't tell us anything about the state of an assumption.

Anyway, from reading the literature on theories (see my previous blog posting), I quite like the idea of having assumptions tell us something about the code, that those assumptions are validated.

The syntax of my addin is quite poor, and there's not really enough validation of user input, but I'm aiming to try to do some theory-driven development (theorizing?) using it, to see what feels good and what grates.

Any feedback gratefully received (especially - is it valid to say that this is an implementation of a Theory, are validation of assumptions useful or unnecessary fluff?)

Here is the syntax of my extension.

 [TestFixture]
 public class TheorySampleFixture
 {
     [Theory]
     [PropertyData("MyTestMethodData")]
     [InlineData("Parity", new object[] { 1.0, 1.0, 1.0 })]
     [InlineData("Parity 2", new object[] { 2.0, 2.0, 1.0 })]
     [InlineData("Double Euros", new object[] { 2.0, 1.0, 2.0 })]
     // This does not match the assumption, and will cause this
     //specific theory Assert to fail, in which case we will get a pass overall.
     // If the unit under test were changed to somehow handle zero exchange rate,
     // the body of the theory method would pass, but the
     // assumption would still not be met and overall we will register a failure.
     [InlineData("ExchangeRate Assumption Check", new object[] { 2.0, 1.0, 0.0 })]
     // This case will fail, there is an assumption that the dollar value is not three,
     // but passing in a value of 3 doesn't cause a failure in the code, demonstrating
     // that the assumption serves no purpose
     [InlineData("This should fail, assumption met but no failure in method", new object[] { 3.0, 1.0, 3.0 })]
     [Assumption("ConvertToEurosAndBackExchangeRateIsNotZero")]
     [Assumption("DollarsNotThree")]
     public void MyTheoryCanConvertToFromEuros(double amountDollars, double amountEuros, double exchangeRateDollarsPerEuro)
     {
         // Should check are equivalent within a tolerance
         // Calls static method on Convert method
         Assert.AreEqual(amountDollars, Converter.ConvertEurosToDollars(Converter.ConvertDollarsToEuros(amountDollars,
         exchangeRateDollarsPerEuro), exchangeRateDollarsPerEuro));
     }

     // Assumption is that the exchange rate is not zero
     public bool ConvertToEurosAndBackExchangeRateIsNotZero(double amountDollars, double amountEuros, double exchangeRateDollarsPerEuro)
     {
         // Should have a tolerance on this
         return exchangeRateDollarsPerEuro != 0.0;
     }

     // Assume that dollar value not equal to three
     // This is just to demonstrate that an invalid assumption results in a failure.
     public bool DollarsNotThree(double amountDollars, double amountEuros, double exchangeRateDollarsPerEuro)
     {
         return amountDollars != 3.0;
     }

     /// Returns the data for MyTestMethod
     ///
     public IList MyTestMethodData
     {
         get
         {
             List details = new List();
             details.Add(new TheoryExampleDataDetail("Some other case should pass", new object[] { 2.0, 20.0, 5.0}));
             return details;
         }
     }
 }

 public static class Converter
 {
     public static double ConvertEurosToDollars(double amountDollars,
         double dollarsPerEuro)
     {
         return amountDollars * dollarsPerEuro;
     }

     public static double ConvertDollarsToEuros(double amountEuros,
         double dollarsPerEuro)
     {
         return amountEuros / dollarsPerEuro;
     }
 }


A nicer syntax/api would be to have the assumptions inline:


public void CanConvertToEurosAndBack(double amountDollars, double amountEuros, double exchangeRateDollarsPerEuro)
{
Assume.That(exchangeRateDollarsPerEuro != 0.0);
Assume.That(amountDollars != 0.0);

// Checks are equivalent within a tolerance
// Calls static method on Convert method
Assert.AreEqual(amountDollars, Converter.ConvertEurosToDollars(Converter.ConvertDollarsToEuros(amountDollars,
exchangeRateDollarsPerEuro),exchangeRateDollarsPerEuro));
}


Here's the rules of my Theory Implementation

If there is no example data, the theory passes (we may want to change this in the future).
If there are no assumptions for a theory, then each set of example data is executed against the theory each producing its own pass or fail.

If assumptions exist, the each set of data is first validated against the assumption - if it meets the assumption, then the test proceeds and any test failure is flagged as an error.
If the example data does not meet the assumption, then if the test passes it indicates that the assumption is invalid, and that case is marked as a failure, with a specific message "AssumptionFailed". Any assertion failures or exceptions in the actual theory code are treated as passes. (in the future, would we want to mark the specific exception expected in the test methdo if an assumption is not met?).

NOTE: we may want to mark as a failure any theory for which ALL example data fails the assumptions, as a check that the
actual body of the theory is actually being executed. I've not done this for now as it would be trickier with the current
NUnit implementation.

Similarly, I was thinking of failing if any of the assumptions weren't actually executed, but again, this is tricky in the current NUnit implementation (and may not give us much).

Automated exploration would not follow the last two suggested rules. The automation API would need to generate its data and execute it as if it were inline data. It may be helpful for the automated tool to be able to retrieve the user-supplied example data, so it doesn't report a failure for any known case, but this is probably not necessary.

Feedback on these rules would be most welcome. If you want to change the behaviour of the assumptions (i.e. have assumptions only filter and nothing more), then the behaviour can be changed in TheoryMethod.RunTestMethod()

Here's the output of the above theory:

Labels: , ,