OpenCover and FitNesse/fitSharp

OpenCover is a great tool for measuring .NET code coverage. In ATDD some tests are written and documented below system level.
If you use FitNesse/fitSharp the code coverage cannot be determined by calling FitNesse on the console via java -jar fitnesse-standalon -c args.

But... the test runner, Runner.exe, is implemented in .NET.

It is called from the FitNesse server with arguments args1 = -r {assembly list} HOST PORT SOCKET.

You can get a coverage report from OpenCover by defining a new test runner using OpenCover as a proxy. The FitNesse server will call the proxy with args1. The new test runner (definable in the wiki via the global variable TEST_RUNNER) will call OpenCover.Console.exe which then calls the original runner passing args1 on and returning its error code with the -returntargetcode argument.

Breaking Dependency on Third Party Executables - Examples in Go

Often in distributed systems our products depend on third party applications. In this post I describe how we can mock these in order to improve coverage, testability and ease the setup of the test environment.

Imagine you have an application that calls an executable. You could for example create an XML document from a database and feed it to a third party application to get a PostScript file. We implement the Decorator Pattern and use a suitable class or function in our programming environment to

1. Call the executable with or without certain arguments (input interface)
2. Intercept the standard output and error to check our results and return to the main application (output interface)

In .NET we could use the Process class to accomplish this: we wrap the CLI into a Process object and redirect stderr and stdout for verification.

On the unit test level we can mock the decorator itself. But if we have legacy code which isn't unit testable, or at the integration test level, it's useful to mock the third party executable itself. This will:

1. speed up testing in case the third party application would take a while to finish,
2. improve coverage: by taking control over the third party application we can provoke error states and need less test data setup to comply with the interface,
3. make it easier to log which arguments the application is called with.

I will call a mocked executable FakeExe.

If our executable expects certain inputs we can control our FakeExe's behaviour based on these. We create an arguments encoder and a decoder. Here is a short example in Go of how we could control the exitcode for an executable expecting an XML file as input:

 package fakeexe  
 //...  
 func EncodeArgument(exitCode int) string {  
      return strconv.Itoa(exitCode) + ext  
 }  
 type fakeExe struct {  
      ExitCode int  
 }  
 func (f *fakeExe) DecodeArgument(arg string) {  
      var err error  
      if len(arg) > len(ext) {  
           ecode := arg[:len(arg)-len(ext)]  
           f.ExitCode, err = strconv.Atoi(ecode)  
      } else {  
           err = errors.New("input not valid" + arg)  
      }  
      if err != nil {  
           f.handleError(err)  
      }  
 }  
 func (f *fakeExe) Run(arg string) {  
      f.DecodeArgument(arg)  
 }  
 func (f *fakeExe) handleError(error) {   
 //...  
 }  

 package main  
 import (  
       "fakeexe"  
       "os"  
 )  
 func main() {  
       f := new(fakeexe.fakeExe)  
       f.Run(os.Args[0])  
       os.Exit(f.ExitCode)  
 }  

The Encoder can be used in test setup code in order to generate the correct input args for the FakeExe. It can be extended with the following useful behaviour:

1. Write a log: this way we can control that the Decorator calls the executable as expected and we can check the FakeExe in case the possible actions are more complicated than the easy example given above.
2. Use input files from paths, for example if there is an input files directory which will be used as working directory. This can also be used to configure the FakeExe in case of several similar expected behaviours where we might not want to implement a seperate FakeExe for each executable we mock.

At some point I faced the problem that the working directories of the executable wasn't known beforehand, it was created when the task using the Decorator was run. Furthermore, many instances of the same executable were called in a workflow. This created two problems:

1. There was no use in writing the log per FakeExe instance: I needed a log of all instances together.
2. Configuring the FakeExe by a configuration file beside it was undoable because it wouldn't have been copied to the target working directory.

I solved the problem by implementing a logger and a configuration service which in Go reduced to just some lines of code, see http://golang.org/pkg/net/. The easiest implementation might look like this:

 package configuring  
 import (  
      "fmt"  
      "io/ioutil"  
      "net"  
 )  
 var ConfigFilename = "Config.txt"  
 //...  
 type server struct {  
      getListener func(protocol, port string) (net.Listener, error)  
      ln     net.Listener  
      protocol  string  
      port    string  
 }  
 func (srv *server) Start() {  
      var err error  
      srv.ln, err = srv.getListener(srv.protocol, ":"+srv.port)  
      if err != nil {  
           panic(err)  
 }  
      for {  
           conn, err := srv.ln.Accept()  
           if err != nil {  
                fmt.Println(err)  
                continue  
           }  
           go srv.sendConfig(conn)  
      }  
      return  
 }  
 func (srv *server) sendConfig(conn net.Conn) {  
      bytes, err := ioutil.ReadFile(ConfigFilename)  
      if err != nil {  
           panic(err)  
      }  
      if len(bytes) > MAX_MESSAGE_LENGTH {  
           panic("Config message too long.")  
      }  
      _, err = conn.Write(bytes)  
      if err != nil {  
           panic(err)  
      }  
      return  
 }  
 func (srv *server) Stop() {  
      if srv.ln != nil {  
           srv.ln.Close()  
      }  
 }  

 package logging  
 import (  
      "net"  
      "fmt"  
 )  
 //...  
 type server struct{  
      getListener func(protocol, port string) (net.Listener, error)  
      ln net.Listener  
      protocol string  
      port string  
      msgs []string  
 }  
 func (srv *server)Msgs() (msgs []string){  
      msgs = srv.msgs  
      return  
 }  
 func (srv *server)Start() (err error){  
      srv.ln, err = srv.getListener(srv.protocol, ":" + srv.port)  
      if(err != nil){  
           panic(err)  
      }  
      for {  
           conn, err := srv.ln.Accept()  
           if (err != nil){  
                fmt.Println(err)  
                continue  
           }  
           go srv.appendMessage(conn)  
      }  
      return  
 }  
 func (srv *server)appendMessage(conn net.Conn){  
      defer conn.Close()  
      buf := make([]byte, MAX_MESSAGE_LENGTH)  
      msg_length, err := conn.Read(buf)  
      var msg string  
      if(err != nil){  
           msg = err.Error()  
      }else{  
           msg = string(buf[:msg_length])  
      }  
      srv.msgs = append(srv.msgs, msg)  
 }  
 func (srv *server)Stop(){  
      if(srv.ln != nil){  
           srv.ln.Close()  
      }  
 }  

From this the next step could be the implementation of a little DSL for our testing extending the FakeExe with an interpreter like for example in Bob's Blog - Writing a Lisp Interpreter in Go. Then, instead of sending concrete implementation specific configuration values with the configuration service we just send a script:

 package fakeexe  
 import (  
      "lisp" //https://github.com/bobappleyard/golisp  
      "io"  
      "strings"  
 )  
 //...  
 func (f *fakeExe)Run(script string){  
      i := lisp.New()  
      i.Repl(strings.NewReader(script), io.Stdout)  
      //...  
 }  

*Examples are written in Go. 

Agile manual test case creation and management

What happens if you work in an environment where you have to run acceptance tests manually; an environment where you cannot run an automated regression test suite in each iteration and thus can't report automatically which test cases have to be adapted or marked as obsolete?  Is agile even possible without automation?

In this post I try to propose a workflow for agile testers - agile as in agile athlete - to conquer this essential lack of automation using TMS like TestLink  (finally surrendering myself to using the word "agile" in one of my posts). You might ask: "Why is he talking about regression and release test suite and at the same time about agile?" Let me answer this: agile methodologies, e.g. SCRUM, might not always be correctly implemented, and though this might lead to failure (sometimes), testers have to adapt to the environment they test in.

Lifetime of an agile test case

First there are acceptance tests, then these are extended in order to find bugs or pursue other test goals. So we are in an iteration and have a set of test cases but in agile these can have quite different lifetimes. When testing manually we ask ourselves: How are the odds that this test has to be run again, ever? If test cases were automated the cost for running that test case again would be nearly zero. But if checked manually the picture changes a lot. For example:

1. Imagine you develop a CMS. We will want to export the content, like a picture. There are tests for that. But in a later iteration that file will automatically be opened in MSPaint. This additional feature makes the first test set obsolete for manual testing: if the picture isn't opened in MSPaint, we'll check why that is and we'll find or not that the file has been exported before.
2. If there was a dead button in the GUI, the fix should form part of our regression test suite. The lifetime of this test case is undetermined, but for now we know that it will survive until the next regression test.
3. If we change test data or namings in the project, e.g. image vs. picture, the written test case will need refactoring in order to survive.

The important point is that I suppose that testers have developed an intuition about the lifetime of the test cases. Testers assign life expectancies to test cases.

Test cases with low life expectancy (throw-away test cases)

If our intuition tells us that a test case is quite specific and won't probably need to be repeated ever again, it's enough if we write it down quickly and concretely, e.g.

1.) Select a picture.

2.) Click the Button labelled "Download".

3.) Open the documents folder of the user.

=> There is a folder OurCMS. It contains the selected picture.

These test cases can be flagged in the TMS. In TestLink we can assign Test Importance = Low or create a custom field "throw-away". And after the test case ran successfully we Deactivate this version. This way we know, we won't have to care about it again but still we document what has been tested.

Test cases with high life expectancy

If our intuition tells us that a test case describes a general or essential aspect of the functionality of the system, it will have a somewhat higher life expectancy. In our example 1.) above, in the beginning we might be talking about downloading and managing photos; later about graphics or images. Or we might change the default check out location. If we anticipate this change, we can reduce refactoring in future relase or regression tests by writing our test case as mimicking coded tests:

Preconditions:

var GRAPHIC = a picture

var BUTTON = Download

var CHECKOUT_FOLDER = $USER$/Documents/OurCMS

Steps:

1.) Select GRAPHIC

2.) Click BUTTON.

=> CHECKOUT_FOLDER contains GRAPHIC_FILE.

Testers can document decisions they took while testing by specifying the chosen test data easily:

Notes:

GRAPHIC = VacationsExamples/DayOnTheBeach.png

As this test case is probably run again in the future we'll assign a higher prio and avoid deactivating it. Written the test case in this fashion, we'll be able to quickly refactor it when the system has changed by just adapting Preconditions.

Of course, now we talk to humans as if they were computers, but surely this is how non-testers treat us, too, demanding manual testing and high coverage, maintainability and traceability.

Some advanced code testing

For those who don't want or don't have the time to read thick books or framework testing APIs, I have collected a few topics that can help improve your unit testing but might not always be obvious. Note that they're not purely restricted to C# even though the examples are.

Mighty testing frameworks

There are some mighty mocking and unit testing frameworks out there with impressing features like mocking dependent-on static methods, testing private members etc. Although in some cases these are vital, e.g. when writing tests for legacy code, they might lead you to write your code with lower quality taking less care about class design, dependencies etc. What's better: code quality or a new (and possibly costly) dependency on a mighty framework? Code quality does not depend on the language you're programming in, it depends mainly on you as a developer!

Use delegates for dependency to static methods

C# delegates can work like functions in languages like Python or Go where they are first class citizens. This is a good thing, e.g. in the following case:

 public void Init()  
 {  
      var content = File.ReadAllText(this.Path);  
      ...  
 }  

We have a dependency here making testing somewhat difficult. Whereas in some situations it's good pratice to wrap call to static methods into an instance, here we can get along without it:

 public void Init()  
 {  
      this.Init(File.ReadAllText);  
 }  
 internal void Init(ReadAllText readAllText)  
 {  
      var content = readAllText(this.Path);  
      ...  
 }  
 internal delegate string ReadAllText(string path);  

Dependencies to the surface

Somewhat related to the last topic on delegates is the the following: Often we programm quickly and only afterwards discover that we have dependencies, especially to built-ins like System.IO.FileInfo. Now, built-ins can have bugs, too. And a dependency on them should better be injected, e.g. for testability or for extendability. We can avoid missing those dependencies by not using global imports, by deleting any using statement at the beginning of our files and using full namespaces. What is the difference between

 using System.IO;  
 using System.Xml;  
 ...  
 internal void Init()  
 {  
      var file = new FileInfo(this.Path);  
      var dir = new DirectoryInfo(this.DirPath);  
      var xdoc = new XmlDocument();  
      ...  
 }  

and

 internal void Init()  
 {  
      var file = new System.IO.FileInfo(this.Path);  
      var dir = new System.IO.DirectoryInfo(this.DirPath);  
      var xdoc = new System.Xml.XmlDocument();  
      ...  
 }  

The difference is that in the second case you're getting so tired of typing the namespaces that you will want to do something about it. For the moment you cannot use global imports, so you will need to refactor to dependency injection, interface extraction etc.

The protected antipattern

There is this rule that production code and test code should not be mixed. In .NET we choose to create seperate test projects and only test against the public interface, letting private members become internal to be testable if necessary and sensible to do so.
Another possibility is to set protected instead. Then in our test project we just inherit and overwrite.
In my experience though, this second approach has at least two downsides that the use of internal doesn't have:

• the access modifiers totally loose their sense because we cannot control what's done with protected members.
• the inherited class will be tested and in a more complicated setup in the end we might loose track. Did we really test our code or the test code?

Any vs. Some

In order for our test cases to serve as documentation we need method and variable names communicating intention:

 public void TestCaseConstructorSetsProperties()  
 {  
      var to = new TestObject(anyParameter());  
      AssertThatPropertiesAreSet(to);  
 }  

My personal gusto is only to use the prefix any if null is permitted, too. So if - following common practice - the constructor checks for null argument, this will be another test case. We can opt for using anyNonNullParameter() or simply:

 public void TestCaseConstructorSetsProperties()  
 {  
      var to = new TestObject(someParameter());  
      AssertThatPropertiesAreSet(to);  
 }  

Advanced setup and teardown - cleanup files example

Have a look at the following code:

 public void TestCaseUsingFileSystem()  
 {  
      var file = createTestFile();  
      var to = new TestObject();  
      to.doSomething(file);  
      AssertThatSomethingHoldsOn(to);  
      file.Delete();  
 }  

The problem here is that the file probably won't be deleted if the assertion fails, making this test case fragile. Surely, you can think of other objects that might need proper tear down even if the test fails in order to assure correctness of the test fixture. A common solution of this problem is to introduce some class variable serving as trash and using a shared teardown. Mind also the file creation method which simply could have been called createTestFile() as before:

 public void TestCaseUsingFileSystem()  
 {  
      var file = createAndRegisterForCleanupTestFile();  
      var to = new TestObject();  
      to.doSomething(file);  
      AssertThatSomethingHoldsOn(to);  
 }  
 public void TearDown()  
 {  
      foreach(var item in this.trash)  
      {  
           try  
           {  
                var file = item as File;  
                if(file != null) file.Delete();  
                ...  
           }catch(Exception e){  
                reportToTestRunner(e.Message);  
           }  
      }  
 }  
 private File createAndRegisterForCleanupTestFile()  
 {  
      var file = createTestFile();  
      this.trash.Add(file);  
      return file;  
 }       

Event checking

You should always check if events are raised, too! An easy pattern for doing so is this:

 public void TestCaseSomeMethodRaisesEvent()  
 {  
      var eventHasBeenRaised = false;  
      testObject.SomeEventHandler = (sender, args) => eventHasBeenRaised = true;  
      testObject.SomeMethod();  
      AssertThat(eventHasBeenRaised);  
 }  




Android Programming and Testing with ADT

It's been a while I dared to take a look at Android application development using Eclipse. I'm certainly surprised about the high quality tutorials and documentation. It is fun to learn, with the right tools, of course. Also, the developer community hasn't left out the vital testing perspective, delivering automatic test project setup, JUnit extensions (even mocks!). If you want to make your first steps with Android application development and learn how to test it right from the start, I recommend to take the following steps supposing you have some experience with Java, Eclipse and, of course, unit testing:

1. Download the Android ADT Bundle here.
2. Follow the steps for setting it up here.
3. Complete the tutorial Building your First App. I recommend using a real device, not only for performance but it feels great ;-) In case you're working on Linux, you'll probably have to add a rule for udev. This is well documented and can be found at the tutorial. Tip: find your vendorId using lsusb and use MSC as the transfer protocol.
4. Skim through Managing Projects from Eclipse with ADT, Building and Running from Eclipse and Testing Fundamentals, the latter being a fascinating read by itself for testing developers (and suffering testers in automation).
5. Make sure you have the Samples for SDK. If you don't you will download them using Android SDK Manager as described here.
6. Skip the Testing from Eclipse with ADT, and dive directly into Activity Testing Tutorial.

This is a good starting point and fun to do :-)

Documenting smells for untestable .NET code with NSubstitute and NUnit

Retrofitting unit tests is hard work and the knowledge about the code could easily be forgotten until it will be refactored. We want to communicate to the developers (and to ourselves after enough time has passed to forget what we've learned about the code) our testing intentions, how the code could be more testable.

One of the unit testing axioms is: always keep production and test code separate. Now if you program in .NET/C# and create a separate test project, then friend assemblies or this pattern can help you to overcome the visibility problem: if I can only access the public interface of my code, there will be much unaccessible code that I still need to test.

But if you work with legacy code there will likely access much untestable code. Tight coupling, hidden dependencies, implementation dependency and many many others are common problems you would normally conquer by refactoring your code (see e.g. Michael C. Feather´s Dependency Breaking Techniques (Part III of his book)).

Sometimes though, you won't have the permission to change the production code. Sometimes you just need to document the code for future reference and prepare test cases so the knowledge you've built up won't get wasted until the code has been refactored..

In this world of retrofitting tests to legacy code you will often find it useful to document the smells or pathologies that make your testing so hard resp. impossible. Instead of writing bug reports, source comments, etc. you might consider next time the following pattern.

For example, imagine you have the following code you're not allowed to make changes to:

public class HasDependencyToImplementation{
  
  public HasDependencyToImplementation(Class c)
  {
    this.c = c; 
    ...
  }

  public void DoSomethingWithC(){
    this.privateProperty = this.c.DoSomething();
    this.PublicProperty = this.WorkWithPrivateProperty();
  }
  
  private T WorkWithPrivateProperty(){
    ... do something with this.privateProperty ...
  }

}

public class Class{...}

Now, if you need to take control over the method Class.DoSomething, you'll normally be advised to extract the interface of Class. Then the constructor admits passing it a mocked Class instance. But what if you're not allowed to do that? Do you want to call the developer to please refactor that code and check in again? Is your developer even in the mood or has time to listen to your pleas? Let's guess they won't. Then with NSubstitute and NUnit you would normally still be able to write a (failing) test like this:

...
[Test]
public void DoSomethingOnC_ChangesState(){
   var c = Substitute.For<Class>();
   var controllableValue = getValue();
   c.DoSomething().Returns(controllableValue);
   
   var hasDependency = new HasDependencyToImplementation(c);
   hasDependency.DoSomethingWithC();
   var expected = getExpectedFor(controllableValue);

   Assert.That(
       hasDependency.PublicProperty,
       Is.EqualTo(
       expected);
}

This test would normally compile, but by running it, we will get an exception from NSubstitute as Class is not abstract and the method Class.DoSomething isn't virtual, so NSubstitute cannot overwrite it. Our problem is: on one hand we want this test to be written for documentation and future testing, after refactoring is done, on the other hand the refactoring developers won't necessarily be able to interpret the test result as a todo for them and you might forget what the problem was until next time you'll work with the code. A solution to this problem might be the following:

We implement a custom assert on NUnit.Framework.Assert:

public static void HasSmell(int PRIO, string PROBLEM, string PROPOSAL){
        Assert.Fail(
                 helper.GetMessage(PRIO, PROBLEM, PROPOSAL));
}

and implement constants

public static Smells{
   public const string IrritatingParameter = "Irritating parameter";
   ...
}

public static Refactorings{
   public const string ExtractInterface = "Extract interface";
   ...
}

Then we can add documentation for developers:


...
[Test]
public void DoSomethingOnC_ChangesState(){

   Assert.HasSmell(
          PRIO: HIGH,
          PROBLEM: Smells.IrritatingParameter
                  +"\nClass c: cannot be mocked for checking.",
          PROPOSAL: Refactorings.ExtractInterface);

   var c = Substitute.For<Class>();
   var controllableValue = getValue();
   ...
}

Which not only will fail when run by developers, but the NUnit runner will tell them

Test DoSomethiongOnC_ChangesState failed:
SMELL PRIO HIGH
PROBLEM: Irritating parameter
Class c: cannot be mocked for checking.
PROPOSAL: Extract interface

History Based Heuristics for Regression Testing

Common Testing knowledge - see for example this article - recommends you to:

1. Identify test areas of your product by
• checking recent release notes
• checking test cases for internal releases
2. Select and add test cases based on the identified areas
3. Add some basic test cases (some kind of smoke test)

This really is a good way to get the job done, though in a legacy environment where you would expect the testing process to have much less lifetime than your product under test, you might want some more testing to be done.

Software Bugs por FastJack, en Flickr

Imagine Regression Testing as part of a bigger test iteration planned for your next product release. As by 1. through 3. you only can check if recently - since the last release -  modified items continue to work pleasantly, you still encounter yourself with that blind side of all those things that haven't been tested ever.

How to approach? You might want to organise a bug bash. Or you might consider to expand 3. to more test areas. But which would those be? If you work in a legacy environment you probably at least have a bug tracking system (BTS) at your disposal that has lived since the beginning of any of your company's development process (or at least longer than your testing documentation).

The idea is quite simple and is an analogue of the one to identify test items for test driven maintenance.

First we identify our key numbers we can normally obtain by asking the underlying data base of our BTS:

Key	Name	Description
D	Number of Duplicate Bug Reports	Duplicates come up if several users have found an issue in different contextes. The more duplicates a bug report has the more important it is to us to identify the underlying area.
R	Number of Related Bug Reports	The more related bug reports are known the bigger the test area must be and therefore the more important the bug report itself.
C	Number of Comments	Many comments indicate a matter worth a discussion. This number also includes status changes (like New => Confirmed).
S	Summary length	Although a long, unconcise bug summary may be symptom of a bad style, we want to assume that complexer bugs need a longer summary.
B	Description Length	Same as in case of S: we want to assume that more complicated issues need a longer description / report.
P	Priority (mapped to a numeric value)	There might be different priorities (internal, external) but after all there's a reason why someone decided to assign such priority.
T	Creation date	If the first bug report ever has status fixed and after n years the bug hasn't come up again, we assume the feature to be stable. Also, if we tested a feature in our last internal release test ("recent date"), we might not need to re-test it again.

From all those numbers we can derive values indicating the relevance for our regression test.
Assuming linear relations we might calculate then numbers for each bug report {n_D,...,n_T} in [0,1] like this:

• n_D = D/(maximal number of duplicates of any bug report)
• n_R = R/(maximal number of related bug reports of any bug report)
• n_C = C/(maximal number of comments of any bug report)
• n_S = S/(maximal number of characters in trimmed string of the summary of any bug report)
• n_B = B/maximal number of characters in trimmed string of the description of any bug report)
• n_P = P/(maximal possible priority)
• n_T = ticks(T)/ticks(recent date) or 0, if the report was created after the set recent date.

We don't have much experience about which of those numbers might be the most important and we don't want to make a scientific study of it, so we set weights {w_C,...,w_T} being numbers in [0,1] such that w_C + ... + w_T = 1. These weights allow us to play with how much importance we attribute to each key number. For example, if we only wanted to evaluate our bug reports based on the creation date we would set w_C = ... = w_P = 0, w_T = 1 meaning that we don't care about the other key numbers.

For any selection of these weights we then calculate for each bug report its relevance as the weighted mean

K(bug report) = (w_D * n_D + ... + w_T * n_T)


K is now a good mean to identify bug reports which matter most to us regarding our intuitions and how much we think we should take each factor into account. What do you think?