Levels of automated testing within a single application

We need a common language for the different types of automated testing.  We’re partially there, but the term “unit test” is still very confusing.  Here, I’ll lay out the different types of automated tests I find helpful with a single application:

  • Unit testing – testing a single class or possible a small group of collaborating classes (absolutely does not call out of process and is the fastest-running of all automated tests).  Running 1000 unit tests in 3 or 4 seconds is common.
  • Full system tests – through the UI integrated with the full application including the database.  May or may not use real system dependencies such as external web services.  (These are the slowest of all tests)
  • Integration testing.  Here, there are some categories.
    • Data access tests.  Used to test repositories, data access classes, etc.  These tests validate the translation from entities to data.  These tests run all SQL and test the structure of the database schema as well.  A real database must be involved.
    • General scenario testing.  Any time it’s appropriate to pull a section of the application in and run a lot of classes together, this is an integration test.  It involves several parts of the system, not just one.  It can run fast if completely in process, or it can be slow if it requires an out-of-process call such as leveraging the file system.

This is not an exhaustive list, but it includes most of the automated testing on a typical enterprise application.  Feel free to comment with any type I may have left out.

Scott Bellware reasoned that the database needs to be left out for unit testing.  I completely agree.  Unit testing, by common definition, excludes external dependencies.  It’s not a unit test if we reach out and touch things.  When you have the right number of unit tests (for example, I’ve worked on a smart client system with 80,000 lines of code and 1300 unit tests and another 700 integration tests), you can’t afford to take more than a few milliseconds to run each one.  You need your unit tests to run very quickly.  Otherwise, you won’t run them very often.

Conversely, this doesn’t mean that the database should be ignored when testing a system.  There are plenty of reasons why a database, SQL, or stored procedures, triggers (shudder), views, etc can cause a bug in the system.  I insist writing an automated integration test for every database operation.  How else can we verify that the database operation works correctly?  We can’t.  It is important, however, for communication’s sake, to understand that these database-inclusive tests are integration tests, as are any tests that exercise an external dependency.

Automated testing with the database REQUIRES the following:

  1. Every developer has a dedicated instance of the database that can be dropped and created at will.
  2. Tests must be responsible for their own data setup.  An empty database should be all that is required to run the test.  The test must be responsible for adding data for the appropriate scenario before testing the scenario.
  3. You will want to generalize test data setup because it isn’t feasible to expect EVERY test to set up all the data.  A general data set that sets a base line of data is very useful and can be invoked with a data helper class.  Then each test can just add specific data necessary for it’s test case.
  4. Data setup, database creation, etc should be automated.  If it’s manual, it cost more, and you won’t run the tests as often.
  5. Database schema must be in source control with the code.  Without that, you never know what the correct version of the schema is.

Another of Scott’s points: “As a side effect of doing the necessary dependency injection, you often get a cleaner and more explicit separation of concerns – which makes software easier to change and maintain.”

He’s right.  If you can’t unit test your domain classes because everything you do with them requires a real database to be online, you have an indication that you aren’t separating concerns.  Data access should be independent of domain object behavior in most cases.  I should be able to verify that a Customer object can Sort() itself without invoking a database query, but if constructing a Customer initiates a database call, my domain model is then materially coupled to the database and needs to be separated.

Jeremy Miller is of the same mind in his comment: “Referential integrity, non null checks, and sundry other data constraints.  All good things.  All a pain in the ass when you’re unit test only needs a single property set on the InvoiceItem class.”

To help clear up some confusion with the term “unit test”, I propose a simple constraint in our dialog:  If the test calls out-of-process, it is then disqualified from “unit test” status and falls into “integration test”.  Feel free to argue in the comments. 🙂

Speaking at AlamoCoders user group in July

If you are in the area, come on over to the AlamoCoders group in San Antonio, TX.  AlamoCoders is a relatively new .Net User Group in San Antonio.  Several founding members attended the Austin Code Camp in May.

I’ll be speaking on July 9th on the following (same session that got great reviews at the Austin Code Camp):


Fundamentals of modern programming – Join Jeffrey Palermo for a talk that gets back to the basics of programming with .Net in the 21st century. This talk will not venture into Test-Driven Development, Domain-Driven Design, O/R Mapping or other recent advances in programming. Instead, I’ll focus on fundamentals that few colleges or training courses are equipped to cover. Some fundamentals include the actual differences between procedural programs, object-oriented programs. Code presented herein will be basic and geared toward illustrating specific fundamentals. An example presented will be what a class should look like and how to decide when to create a class. If you feel a bit rusty on your fundamentals or if you’d just like a refresher course, this talk is for you.

Modify and restore Firefox extensions that fail to load

Recently, my Firefox browser started loading without the extensions I’ve installed.  I run my Firefox browser off of my 4GB Cruzer using the PortableApps suite.

When Firefox loaded, it would load without a single extension, but when I opened the “add-ins” box, it listed all of my extensions.

Here is the thread where I found my particular solution.  I was able to get Firefox to load my extensions again by modifying the files within my Firefox profile (find it in you user account appdata folder).  I deleted extensions.cache, extensions.ini and extensions.rdf.  When I started Firefox the next time, it regenerated these files using the extensions that were in my profile (folders like {2fa4ed95-0317-4c6a-a74c-5f3e3912c1f9}).

Problem solved for me.  I did take notice that Firefox is blazing fast on start-up. . . until I load a ton of extensions.  With the number of extensions I have, it’s more like Visual Studio since I’m loading up a whole suite of web development tools.  Firefox is just as much a web IDE as Visual Studio – including the RAM usage.

I’m the newest Solutions Architect MVP

In 2006, I was awarded as a C# MVP and was renewed for 2007.  Now, I’m a Solutions Architect MVP.  I look forward to providing more feedback to Microsoft’s Architecture Strategy team in the future.

I’d like to thank Simon Guest and my new MVP lead Steve Dybing.  Also, thank you, Rafael, for being a great MVP lead for the C# track.

My new MVP Profile

Xml is the code of the future – so long C# – say it isn’t so.

Ok, so the title is a bit sarcastic.  I remember the debut of Xml.  We used it for data.  It was a way to improve on comma-delimited strings.  We converted from flat files to xml files for data.

It wasn’t long before scripting could be expressed in Xml, and now I see teams with large libraries of executable Xml in the form of build scripts with NAnt.

I think it’s going to far, however.  I’m beginning to hear that all software can now be expressed in Xml if you have the right tools.  The implications of that is just shifting from C# programming to Xml programming.  Whatever the language of execution is – that’s the programming language.

I share Ayende’s concern for Xml programming.  Xml is not a 5th generation language.  Just because designers can generate Xml instead of C# doesn’t mean code is going away.  It just means that the chosen syntax is different for the generated code.

My stance on code generation has always been:

  • Machine generates code, human maintains (bad)
  • Machine generates code, machine runs code, human never has to see code (good). i.e. C# to MSIL
  • Machine generates code exactly how human would have written it anyway (good – just saves typing).
  • Machine generates code, human has to modify code and then maintain (worst)

Designers fall into the category that WANTS to be the 2nd bullet point, but that never happens.  WinForms tried to do this with the .designer file, but that code still needed to be understood and tweaked from time to time.  Designers don’t have good track records because they always constrain flexibility.  When you hit a wall, you always have to fall back and change the generated code.  That’s the problem.  Good intentions, but it won’t happen that way.  For it to work, the designer has to be the language. 

Designers come and designers go.  Languages stay and evolve.

Resharper (R#) 3.0 has been released!

Version 3.0 of my favorite IDE add-in has been released.  If you purchased v2.5 recently, you get a free upgrade**

If you are unfamiliar with the tool, check out my previous posts on it:


** All those who have bought ReSharper 2.5 since April 15, 2007 qualify for a free upgrade!

Another free unit testing add-in for Visual Studio 2005

There are plenty of ways to run your NUnit tests while developing and during the build.  JetBrains has released a unit test running add-in for Visual Studio, and they are giving it away for free.  It’s called UnitRun.  The screenshot to the right shows how you can execute a test from the margin.  I like to map CTRL+T as my shortcut to run a test.




You can also run all of the tests in your solution with a larger window.  (above)  This isn’t a new release.  It’s been out for some time, but I’d never mentioned it.  I use unit run.  It’s actually a feature of the larger Resharper package.

Other notable unit test runners are:

Resharper 2.5/3.0 hack to speed up CTRL+N type discovery

When R# 2.5 came out, I noticed that I had to stop typing briefly after pressing CTRL+N before the filtered list of types was displayed.  I was a bit annoyed because version 2.0 showed the list instantly.  From a few emails, I learned that this was an intentional feature and that the delay was not made configurable but might be in the future.

I’ve just downloaded v3.0 RC, and the delay is still there.  I raised the issue on the forum, and I found a nice workaround:

  • Go to the roaming users application data folder (C:Users{my user}AppDataRoamingJetBrainsReSharperv3.0vs8.0) on Vista
  • Open UserSettings.xml
  • Find the node: (xpath) /Configuration/SettingsComponent/integer/setting[14]
    • You’ll notice the delay is “300”.  Change it to <setting name=”UserInputDeferTime”>1</setting>
    • This will drop the delay from nearly 1/3 of a second to 1 millisecond.
  • Save the file.

Don’t put “0” because it will be ignored.  The end result is that you have to delay a tiny/tiny bit, and this feature is much more usable.

For those who don’t know what CTRL+N is, here is what it looks like (old screen shot)