Automated testing with .Net – an overview – level 200

Reality

In reality, developers don’t like to do much testing.  Developers aren’t testers.  We typically will write code while making
certain assumptions about variables and, objectively knowing the expected behavior
of the syntax, we might run it once and call it done.  Typically, bugs hide where the code wasn’t
rigorously tests or in paths that weren’t tested at all.  I think everyone agrees that without testing,
the software will have bugs (and often even after testing).  I heard of a crazy management quote that is
very sad: “If you have to test it, you aren’t building it right.”  I sure am glad that manager wasn’t involved
with automobile development!  In reality,
we need testing

 

Types of common
developer testing

  • Actually
    running the code written
  • Simple
    console application or winforms test harnesses
  • Running
    the application through the UI locally or on a development server.

 

Types of testing

  • Unit
    testing
  • Integration
    testing
  • Acceptance
    testing
  • Load
    testing
  • Performance
    testing
  • Security
    testing
  • Exploratory
    testing
  • And
    many more

 

Approaches

  • Do all
    manual testing with or without the help of small tools.
    • With
      this approach, the cost is high because for every run of the test, human
      time is required.  For every
      release, every test case must be repeated.
    • This
      approach doesn’t scale.  When more
      testing is needed, that directly calls for more human time.  Human time is very expensive compared
      to computing time.
  • Automate
    all testing.
    • The
      cost is low for about 80%, but then goes up.
    • Some
      types of tests are hard or impossible to automate effectively, such as
      security, exploratory, and concurrency testing.  This type of testing needs more human
      attention and isn’t easily repeatable.
    • To
      fully succeed at this approach, testers need to be highly skilled at
      scripting.
  • A
    pragmatic approach – automate the testing that is easiest.
    • Unit
      testing
    • Integration
      testing
    • Acceptance
      testing

Unit testing

What is a unit test? 
From Wikipedia:  A “. . . unit
test
is a procedure used to validate that a particular module of source
code is working properly”.  A unit test
is a very unique type of test with some fundamental constraints.  If a test violates a constraint, it ceases
existence as a unit test.

 

Characteristics of a
unit test

  • A unit
    test isolates and tests a single responsibility.
  • A unit
    test isolates a failure.
  • A unit
    test simulates other collaborators using various methods such as fake
    objects.
  • A unit
    test can run and pass in any environment.

When an automated test reaches out to a file, registry key,
database, web service, external configuration, environment variable, etc, it
becomes an integration test and is polluted with dependency on its
environment.  At that point, the binary
can no longer be sent to another location with the expectation that test
continue to pass. 

 

Unit testing is hard. 
It requires constant vigilance to ensure application code is
loosely-coupled.  Because a unit test
must test a piece of code in isolation, the code must be designed so that
collaborators can be easily separated. 
In short, calling out to a static method from inside a constructor is a
tight-coupling.  This type of code will
hinder unit testing.

 

Integration testing

What is integration testing? 
An integration test is a developer test that combines several units of
code and tests the interaction among them. 
A good integration strategy will include tests along all integration
boundaries.  If a unit of code is tested
at the unit level and then tested during interaction with all collaborators,
the developer has confidence that the code will work well when the entire
system is assembled.  Integration
testing
is the second step in a “test small, test medium, test large”
strategy.  Larger integration tests are certainly useful, but
large tests alone don’t help to pinpoint a problem if they fail. 

 

Characteristics of a
good integration test

  • It
    only includes a few classes.  It
    typically focuses on a single class and then includes collaborators.  If a class has 3 collaborators, the test
    would include 4 classes (the first class and its 3 collaborators).  This test would aim to fake the
    collaborators twice removed so that the focus on the test can remain
    narrow.  This is a subjective rule,
    and the developer should use good judgment regarding how many classes to
    include in an integration test.
  • It is
    completely isolated in setup and teardown. 
    Should not be environmentally sensitive.  If the test requires a file to be on the
    drive, then setup for the test should place the required file there
    first.  If the test needs data in a
    database, the setup should insert the required records first.  It should not depend on any environment
    setting being present before the test is run.  It should run on any developer
    workstation.
  • Must
    run fast.  If it’s slow, the build
    time will suffer, and you will run fewer builds.
  • It
    must be order-independent.  If an
    integration test causes a subsequent test to fail, it is indicative of a
    test not owning its own environmental setup.

 

Acceptance testing

What is acceptance testing? 
An evaluation of acceptance criteria on the system to ensure the system
meets the customers’ needs.  Before a
piece of software can be written the software team must have acceptance
criteria.  This is often called
requirements.  Acceptance testing seeks
to execute the requirements on the system for a pass/fail result.

 

Characteristics of a
good acceptance test

  • Easily
    understood by non-developers as well as the customer.
  • Easily
    created by non-developers as well as the customer.
  • When
    it passes, the customer is assured that the system behaves as needed.
  • Very
    expressive using jargon of the business domain.
  • Is
    repeatable.

 

Automated acceptance testing is the final big gain in the
suite of automated tests.  The acceptance
tests actually define requirements in an objective and executable way.  If a tester finds a bug, he can write an
acceptance test that fails in the presence of the bug.  When the bug is fixed, the test will
pass.  Acceptance tests become an
important part of a regression test suite over time.

 

Stay tuned for more
detailed information on the following testing topics:

  • Unit
    testing
  • Integration
    testing
  • Acceptance
    testing