On my team, we have FitNesse tests surrounding our system.  We’ve integrated them into our cruise build as well.  We developed custom Xsl sheets so that the cruise build report includes fitnesse information.  We even have our regression FitNesse suite fail the build if any FIT tests fail.  When this happens, the cruise build report shows us the actual FitNesse test that failed with the actual fixture tables.  It’s a great way to report this status.  If you are interested in this technique, my colleague, Steve Donie has posted the Xsl stylesheet on his blog.

The FitLibrary has been recently ported to .Net, and I love using the DoFixture.  One problem is that the normal DoRow and DoCell methods of the Fixture class don’t run in the DoFixture.  With other fixtures, I’m used to hooking into to these places in order to spread logic throughout my FIT test.  DoFixture uses two new methods: 
System.Collections.IEnumerable ParameterCells(CellRange theCells)
and
System.Collections.IEnumerable MethodCells(CellRange theCells)

This is how the DoFixture implements it’s special behavior.  For example:
|MyDoFixture|
|Run System|
|Make Sure File|taxes.txt|Was Saved|

This is a simple fixture that makes sure the system saves a file “taxes.txt”.  Here, I’m not using the built-in Check method.  I’m implementing one myself.  Either the file was saved or it wasn’t.

My underlying fixture will look like this:

This works, but will throw an exception instead of failing the test.  I need to register a FIT failure and not an exception.  FIT treats these two differently.  In order to fail a FIT test, I have to use the Wrong(Parse theParse) method.  I have to have the current Parse object in order to call this method.  The Parse object is the current cell/row/table.  The FIT table is composed of a Parse hierarchy.  I don’t really like the way it’s set up, but that’s not the point of this post.

In order to call the Wrong(…) method to fail the test correctly with a reason to output to the screen, I must obtain the current Parse object.  Because the DoFixture doesn’t fire the virtual DoCell or DoRow method, I can’t use these.  Instead, I’ll override a DoFixture method that fires for every row and modify my fixture like this:

With this code, I have my fixture maintain the current Parse (I’m tracking the current row) so that when a line fails, I can properly fail the test with an appropriate message.  This is a much better solution that throwing an exception.  Throwing an exception means that the test environment blew up and needs to be fixed.  Test failures mean the code broke.  An exceptioin means the FitNesse server broke..

Sam Gentile reminds us of the value of FIT when requirements might be vague _and_ complicated.  His team is using it to help the customer understand what they want.  His post reminded me of the value I’m already starting to take for granted.

My team is still early on in our FIT implementation.  We have a few components covered so far, but recently, we created a new service that runs all the time and does stuff.  Do you love the vague description?  It responds to a queue:  One component drops off some work, and it picks it up and does it’s thing.  We have this implemented as a Windows service, so it would be very difficult to test by manual means.  The tester would have no UI at all, and they would have to arrange for work to be dropped off, wait a bit and check that the work was completed. 

FIT enables a whole new dimension to testing this service.  Our tester has been creating his test cases even while we’ve been creating the FIT fixture for it.  We hashed out what the fixture would look like on the whiteboard, and then started.   We have our first draft of the FIT fixture working, and it’s a big eye-opener to be able to test a component (that has no UI) with a GUI.  These are automated tests as well (that’s FIT’s nature), and we’ll run them forevermore.  Once we have these tests passing, they will serve as a regression tests suite for this component.  When a bug arrises, we’ll create a FIT test to isolate the bug and then fix it. 

We’re using the FitNesse wiki to organize our FIT tests.  We’ve set up a special CC.Net build that detects wiki changes and commits those to Subversion.  Our CC.Net build for the component pushes the latest bits to the FitNesse wiki so the FIT tests are always run against the current build.  Read how we integrated FIT into our build here.  Read how we versioned the wiki here.  Mike Roberts has done something similar as well.
 
For this work, we’re using the newly-ported FitLibrary‘s DoFixture.  The DoFixture allows a very intuitive layout of the test.  For more information, I’d recommend reading the FIT book by Ward Cunningham and Rick Mugridge.

A while back, I posted about how my team integrated and versioned our FitNesse wiki with CruiseControl.Net and Subversion.  Mike Roberts found it helpful for creating a similar solution for his team.  He’s shared his experience on his blog.  Hopefully others will find it useful.

For those who don’t know about FitNesse (or Google), Fit is a framework
for creating system-level acceptance tests using Excel worksheets or
Html tables.  FitNesse is a wiki that provides a UI to Fit for
maintaining the acceptance test tables in a hierarchical wiki
website. 

My team uses FitNesse tables to allow testers and product managers to
exercise our entire system (or entire subsystems) through their
tables.  This is far more flexible than the application’s UI, and
it allows for more exploratory testing.

The greatest strength of FitNesse acceptance tests is that they are
executable requirements.  When all the acceptance tests pass, we
know we are done.  If a bug surfaces, we write an acceptance test
to describe the bug and we keep it in a large suite that become strong
regression tests.  There is no mistake about the difference in a
bug an a missed requirement as well.  If it doesn’t have an
acceptance test, then it isn’t a requirement. 

Finally, if you’d like to learn more, here’s a great google search.

You might have notice in the FitNesse urls, a “?test” at the end when you run a test page.  Also, when you edit a page, you are using “?edit”.  There are many of these querystring switches you can use to do some interesting things.  You can see all of these by opening up the Java source:

  addResponder(“edit”, EditResponder.class);
  addResponder(“saveData”, SaveResponder.class);
  addResponder(“tableWizard”, TableWizardResponder.class);
  addResponder(“search”, SearchResponder.class);
  addResponder(“searchForm”, SearchFormResponder.class);
  addResponder(“test”, TestResponder.class);
  addResponder(“suite”, SuiteResponder.class);
  addResponder(“proxy”, SerializedPageResponder.class);
  addResponder(“versions”, VersionSelectionResponder.class);
  addResponder(“viewVersion”, VersionResponder.class);
  addResponder(“rollback”, RollbackResponder.class);
  addResponder(“names”, NameWikiPageResponder.class);
  addResponder(“properties”, PropertiesResponder.class);
  addResponder(“saveProperties”, SavePropertiesResponder.class);
  addResponder(“whereUsed”, WhereUsedResponder.class);
  addResponder(“refactor”, RefactorPageResponder.class);
  addResponder(“deletePage”,DeletePageResponder.class);
  addResponder(“renamePage”, RenamePageResponder.class);
  addResponder(“movePage”, MovePageResponder.class);
  addResponder(“pageData”, PageDataWikiPageResponder.class);
  addResponder(“createDir”, CreateDirectoryResponder.class);
  addResponder(“upload”, UploadResponder.class);
  addResponder(“socketCatcher”, SocketCatchingResponder.class);
  addResponder(“fitClient”, FitClientResponder.class);
  addResponder(“deleteFile”, DeleteFileResponder.class);
  addResponder(“renameFile”, RenameFileResponder.class);
  addResponder(“deleteConfirmation”,DeleteConfirmationResponder.class);
  addResponder(“renameConfirmation”,RenameFileConfirmationResponder.class);
  addResponder(“raw”, RawContentResponder.class);
  addResponder(“rss”, RssResponder.class);
  addResponder(“import”, WikiImportingResponder.class);
  addResponder(“files”, FileResponder.class);
  addResponder(“shutdown”, ShutdownResponder.class);
  addResponder(“format”, TestResultFormattingResponder.class);
  addResponder(“symlink”, SymbolicLinkResponder.class);

Some of the more interesting ones I like are rss, fitClient, and raw.  Yes, “shutdown” will actually shut down the wiki.  Use with care.

The “rss”  querystring parameter will give you an rss feed, and the “raw” parameter will spit out the exact contents of the underlying “content.txt” file.  This is the exact wiki text that the user entered.  “fitClient“ is the one that interests me most because it will spit out the html tables needed to actually run the FitNesse tests offline or in a debugger.  The html is rendered to the browser, but there is a bug in the FitClientResponder class that violates a rule of an Http request, so I can’t programmatically get this page with .Net code (say, the WebClient class). 

Something very interesting I’ve done with the “raw” switch is to pull it back into an NUnit helper class, run it through some simple translator code:

                  StringReader reader = new StringReader(wikiText);

                  string line = reader.ReadLine();

                  while(line != null)

                  {

                        line = line.Trim();

                        if(line.StartsWith(“!”))

                        {

                              string row = line.Trim(‘!’, ‘|’);

                              this.StartTable(row);

                        }

                        else if(line.StartsWith(“|”))

                        {

                              // get rid of leading and trailing “|”, and split on the rest.

                              string[] columns = line.TrimStart(‘|’).TrimEnd(‘|’).Split(‘|’);

                              this.AddRow(columns);

                        }

                        line = reader.ReadLine();

                  }

                  reader.Close();

 

Then, I have the html tables that allow me to create a new Parse object by passing in this html string.  Then it’s trivial to execute this against the FitNesse test engine.

By the way, Google Desktop 2 is great.  I’ve been using it to search and find source code.  Even if the documentation is complete, or you want to dig more, Desktop search is great for finding tidbits in the source code itself (I’ve used MSN search as well.  They both work well).

Follow this post to easily version your FitNesse wiki acceptance tests (or any other tree of files).

The product manager and his crew will use the FitNesse wiki to author and maintain acceptance tests for the system.  These tests are stored in files on the FitNesse server.  This presents a unique problem of backing up and versioning the tests.  The wiki itself supports 15 levels of modifications, but we use Subversion as the standard versioned backup at the company, so we’d like the acceptance tests to go there.

Here is how we solved this problem:

We created a new project in CruiseControl.Net to watch the wiki.  Call it WikiWatcher.  This project uses “filesystem” as it’s source control to watch.  We point this at the FitNesse directory, and CC.Net will react when any of the files there change.  CC.Net then kicks of a Nant script, and that gives us all the power we need.  In Nant, we run two <exec/> task.  The first adds any files not currently in SVN, and the second commits the files.  NantContrib only provides update and checkout tasks for SVN, so we had to hit svn.exe directly with some command line arguments.  This system works well, and within 60 seconds of a Wiki modification, the changes are versioned and backed up in SVN.  (The CruiseControl.Net documentation can fill in any gaps you may have)

Here is the CC.Net config section that defines our project:

<!– this project watches the fitnesse wiki and backs things up in subversion –>
  <project name=”WikiWatcher”>
    <webURL>http://<serverName>/ccnet/Controller.aspx?_action_ViewProjectReport=true&amp;server=aufile01&amp;project=FitNesseWatcher</webURL&gt;

    <triggers>
      <intervalTrigger seconds=”60″ />
    </triggers>

    <modificationDelaySeconds>15</modificationDelaySeconds>
    <publishExceptions>false</publishExceptions>

    <labeller type=”defaultlabeller”>
      <incrementOnFailure>false</incrementOnFailure>
    </labeller>

    <sourcecontrol type=”filesystem”>
      <repositoryRoot>C:fitnesse</repositoryRoot>
      <autoGetSource>false</autoGetSource>
      <ignoreMissingRoot>false</ignoreMissingRoot>
    </sourcecontrol>

    <tasks>
      <nant>
        <executable>c:<some path>trunkbinnantNAnt.exe</executable>
        <baseDirectory>c:<some path>trunk</baseDirectory>
        <buildFile>WikiWatcher.build</buildFile>
        <targetList>
          <target>commit</target>
        </targetList>

        <buildTimeoutSeconds>1500</buildTimeoutSeconds>
      </nant>
    </tasks>

    <publishers>
      <xmllogger />
    </publishers>

  </project>

This is pretty simple.  Just add the above as a CC.Net project node, and you’re off to the races.  The Nant script is even simpler.
Here’s the Nant script:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<project name=”WikiWatcher” default=”commit” xmlns=”http://nant.sf.net/release/0.85-rc3/nant.xsd“>
 <property name=”baseWikiDir” value=”C:fitnesse”/>
 <target name=”commit” description=”Commits entire wiki directory to source control”>
  <fileset basedir=”${baseWikiDir}” id=”wikiFiles”>
   <include name=”**”/>
  </fileset>
  <echo message=”My build ran”></echo>
  <exec program=”c:program filessubversionbinsvn.exe”
        commandline=”add –force *.*”
        workingdir=”${baseWikiDir}” />
  <exec program=”c:program filessubversionbinsvn.exe”
        commandline=”ci -m&quot;automatic checkin&quot; –username <some user> –password <some password>”
        workingdir=”${baseWikiDir}” />
 </target>
</project>

This adds and commits all FitNesse files to SVN, and we’re done.

I’m doing a lot with FitNesse lately, and it’s going quite well.  Unit testing ensures that each component is doing what it is supposed to, and integration testing ensures that the components work well together.  Acceptance tests are a bit different.  They actually test that the software does what the customers think it’s doing.  They test that the developer understood the business need.  They give the customer (or product manager) assurance that the system actually works.

With FitNesse, our product manager (and crew) can  use a simple wiki to exercise the system.  The developers create test Fixtures that speak in the domain language but actually exercise the system under test.  A user may type:

!|fit.ActionFixture|
|start|Emailer|
|press|SendDefaultEmail|
|check|NumEmailsSent|

This simple table is enough to actually exercise the system under test and get it to do something.  If all is well, the Emailer will send a default email.  If this test fails, it saves the company from a black eye when a user encounters the problem.

It was a little tricky to get the FitNesse test integrated with our CC.Net build.  We wanted the acceptance test to be a current status report of where we are.  When all the acceptance tests are passing, we are done.  Consequently, we didn’t want failing acceptance tests to break the build.  Only NUnit tests break the build because they should _always_ be passing.  No problem.  We create a new target with some exec tasks:

<exec program=”<path to TestRunner.exe>” commandline=”-results <some directory>FitNesse-Results.html <FitNesse server> <port> <TestSuite>” failonerror=”false”/>

This will actually call the FitNesse server (with the wiki on it) and execute all the acceptance tests.  This will produce a file in the raw FitNesse format.  We’d like it in xml so we can incorporate it in the CC.Net build report.  Xml transforming isn’t a part of the .Net port, but it is in the Java version, so we’ll just use it.

<exec program=”java.exe” commandline=”-cp binfitfitnesse.jar fitnesse.runner.FormattingOption <some directory>FitNesse-Results.html xml <some directory>Fit-Results.xml <FitNesse server> <port> <TestSuite>” failonerror=”false”/>

This hooks into the transform logic on the FitNesse server to change our raw output file to a nice Xml format.  Now we have something that CruiseControl.Net can use for a build report and email.  We will have to make an Xsl, though.  Here’s a simple Xsl that will pull out the FitNesse summary information and through it out to your CC.Net build summary:

<?xml version=”1.0″?>

<xsl:stylesheet

xmlns:xsl=”http://www.w3.org/1999/XSL/Transform&#8221; version=”1.0″>

<xsl:output method=”html”/>

<xsl:variable name=”fit.result.list” select=”//testResults/result”/>

<xsl:variable name=”fit.wrongpagecount” select=”countU$fit.result.list/counts/wrong[text() > 0])” />

<xsl:variable name=”fit.ignorespagecount” select=”countU$fit.result.list/counts/ignores[text() > 0])” />

<xsl:variable name=”fit.exceptionspagecount” select=”countU$fit.result.list/counts/exceptions[text() > 0])” />

<xsl:variable name=”fit.correctpagecount” select=”countU$fit.result.list/counts)” />

<xsl:variable name=”fit.correctcount” select=”//testResults/finalCounts/right”/>

<xsl:variable name=”fit.failures” select=”//testResults/finalCounts/wrong”/>

<xsl:variable name=”fit.notrun” select=”//testResults/finalCounts/ignores”/>

<xsl:variable name=”fit.exceptions” select=”//testResults/finalCounts/exceptions”/>

<xsl:variable name=”fit.case.list” select=”$fit.result.list//test-case”/>

<xsl:variable name=”fit.suite.list” select=”$fit.result.list//test-suite”/>

<xsl:variable name=”fit.failure.list” select=”$fit.case.list//failure”/>

<xsl:variable name=”fit.notrun.list” select=”$fit.case.list//reason”/>

<xsl:variable name=”colorClass”>

<xsl:choose>

<xsl:when test=”$fit.exceptionspagecount > 0″>fiterror</xsl:when>

<xsl:when test=”$fit.ignorespagecount > 0″>fitignore</xsl:when>

<xsl:when test=”$fit.wrongpagecount > 0″ >fitfail</xsl:when>

<xsl:otherwise>fitpass</xsl:otherwise>

</xsl:choose>

</xsl:variable>

<xsl:variable name=”fit.tests.present” select=”countU//testResults/result) > 0 or count(/cruisecontrol/build/buildresults//testsuite) > 0″ />

<xsl:template match=”/”>

<xsl:choose>

<xsl:when test=”$fit.tests.present”>

<style>

*.fitpass{

background-color: #AAFFAA;

}

*.fitfail{

background-color: #FFAAAA;

}

*.fiterror

{

background-color: #FFFFAA;

}

*.fitignore

{

background-color: #CCCCCC;

}

*.fitheader{

border: solid 1px black;

margin: 1px;

padding: 2px;

}

*.line{

margin: 5px;

}

</style>

<div>

<div class=”{$colorClass} fitheader”>

<strong>FitNesse Summary — Test Pages:</strong> <xsl:value-of select=”$fit.correctpagecount”/> right, <xsl:value-of select=”$fit.wrongpagecount”/> wrong,

<xsl:value-of select=”$fit.ignorespagecount”/> ignored, <xsl:value-of select=”$fit.exceptionspagecount”/> exceptions

<strong>Assertions:</strong> <xsl:value-of select=”$fit.correctcount”/> right, <xsl:value-of select=”$fit.failures”/> wrong,

<xsl:value-of select=”$fit.notrun”/> ignored, <xsl:value-of select=”$fit.exceptions”/> exceptions

</div>

<xsl:for-each select=”$fit.result.list”>

<xsl:variable name=”colorClass”>

<xsl:choose>

<xsl:when test=”counts/exceptions > 0″>fiterror</xsl:when>

<xsl:when test=”counts/ignores > 0″>fitignore</xsl:when>

<xsl:when test=”counts/wrong > 0″ >fitfail</xsl:when>

<xsl:otherwise>fitpass</xsl:otherwise>

</xsl:choose>

</xsl:variable>

<div class=”line”>

<span class=”{$colorClass}” style=”padding:2px;” ><xsl:value-of select=”counts/right”/> right, <xsl:value-of select=”counts/wrong”/> wrong,

<xsl:value-of select=”counts/ignores”/> ignored, <xsl:value-of select=”counts/exceptions”/> exceptions </span>

<span style=”padding:2px;”><xsl:value-of select=”relativePageName”/></span>

</div>

</xsl:for-each>

</div>

</xsl:when>

</xsl:choose>

</xsl:template>

</xsl:stylesheet>

 

Hook this Xsl into your CC.Net configuration, and that’s as hard as it is!  FitNesse results “magically” show up in the build summary.  It’s really a big win for visibility into the state of the software.  The build summary is now the actual status report of the software, and it’s not subjective.  What does 83% actually mean?  With a list of acceptance tests to fulfill, we know exactly when we are done.

 

FitNesse is an acceptance test framework that supports Java and .Net.  When you download it, it will be in Java mode, and all the sample test pages will run Java tests.  It may not be completely clear how to run a new test page against a .Net assembly instead of a Java package.  When you create a new page, add the following:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnetFitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnetfit.dll
!path dotnet*.dll

|eg.Calculator|
|points?|
|false  |

This will run the included Calculator fixture in eg.dll that is included with the download, so you don’t have to write any code yourself.  Note that the FitNesse web server is a Java wiki, so you’ll need the JRE from sun.