Reality

In reality, developers don’t like to do much testing.  Developers aren’t testers.  We typically will write code while making
certain assumptions about variables and, objectively knowing the expected behavior
of the syntax, we might run it once and call it done.  Typically, bugs hide where the code wasn’t
rigorously tests or in paths that weren’t tested at all.  I think everyone agrees that without testing,
the software will have bugs (and often even after testing).  I heard of a crazy management quote that is
very sad: “If you have to test it, you aren’t building it right.”  I sure am glad that manager wasn’t involved
with automobile development!  In reality,
we need testing

Types of common
developer testing

• Actually
  running the code written
• Simple
  console application or winforms test harnesses
• Running
  the application through the UI locally or on a development server.

Types of testing

• Unit
  testing
• Integration
  testing
• Acceptance
  testing
• Load
  testing
• Performance
  testing
• Security
  testing
• Exploratory
  testing
• And
  many more

Approaches

• Do all
  manual testing with or without the help of small tools.
• With
  this approach, the cost is high because for every run of the test, human
  time is required.  For every
  release, every test case must be repeated.
• This
  approach doesn’t scale.  When more
  testing is needed, that directly calls for more human time.  Human time is very expensive compared
  to computing time.
• Automate
  all testing.
• The
  cost is low for about 80%, but then goes up.
• Some
  types of tests are hard or impossible to automate effectively, such as
  security, exploratory, and concurrency testing.  This type of testing needs more human
  attention and isn’t easily repeatable.
• To
  fully succeed at this approach, testers need to be highly skilled at
  scripting.
• A
  pragmatic approach – automate the testing that is easiest.
• Unit
  testing
• Integration
  testing
• Acceptance
  testing

Unit testing

What is a unit test? 
From Wikipedia:  A “. . . unit
test is a procedure used to validate that a particular module of source
code is working properly”.  A unit test
is a very unique type of test with some fundamental constraints.  If a test violates a constraint, it ceases
existence as a unit test.

Characteristics of a
unit test

• A unit
  test isolates and tests a single responsibility.

• A unit
  test isolates a failure.
• A unit
  test simulates other collaborators using various methods such as fake
  objects.
• A unit
  test can run and pass in any environment.

When an automated test reaches out to a file, registry key,
database, web service, external configuration, environment variable, etc, it
becomes an integration test and is polluted with dependency on its
environment.  At that point, the binary
can no longer be sent to another location with the expectation that test
continue to pass. 

Unit testing is hard. 
It requires constant vigilance to ensure application code is
loosely-coupled.  Because a unit test
must test a piece of code in isolation, the code must be designed so that
collaborators can be easily separated. 
In short, calling out to a static method from inside a constructor is a
tight-coupling.  This type of code will
hinder unit testing.

Integration testing

What is integration testing? 
An integration test is a developer test that combines several units of
code and tests the interaction among them. 
A good integration strategy will include tests along all integration
boundaries.  If a unit of code is tested
at the unit level and then tested during interaction with all collaborators,
the developer has confidence that the code will work well when the entire
system is assembled.  Integration
testing
is the second step in a “test small, test medium, test large”
strategy.  Larger integration tests are certainly useful, but
large tests alone don’t help to pinpoint a problem if they fail. 

Characteristics of a
good integration test

• It
  only includes a few classes.  It
  typically focuses on a single class and then includes collaborators.  If a class has 3 collaborators, the test
  would include 4 classes (the first class and its 3 collaborators).  This test would aim to fake the
  collaborators twice removed so that the focus on the test can remain
  narrow.  This is a subjective rule,
  and the developer should use good judgment regarding how many classes to
  include in an integration test.
• It is
  completely isolated in setup and teardown. 
  Should not be environmentally sensitive.  If the test requires a file to be on the
  drive, then setup for the test should place the required file there
  first.  If the test needs data in a
  database, the setup should insert the required records first.  It should not depend on any environment
  setting being present before the test is run.  It should run on any developer
  workstation.
• Must
  run fast.  If it’s slow, the build
  time will suffer, and you will run fewer builds.
• It
  must be order-independent.  If an
  integration test causes a subsequent test to fail, it is indicative of a
  test not owning its own environmental setup.

Acceptance testing

What is acceptance testing? 
An evaluation of acceptance criteria on the system to ensure the system
meets the customers’ needs.  Before a
piece of software can be written the software team must have acceptance
criteria.  This is often called
requirements.  Acceptance testing seeks
to execute the requirements on the system for a pass/fail result.

Characteristics of a
good acceptance test

• Easily
  understood by non-developers as well as the customer.
• Easily
  created by non-developers as well as the customer.
• When
  it passes, the customer is assured that the system behaves as needed.
• Very
  expressive using jargon of the business domain.
• Is
  repeatable.

Automated acceptance testing is the final big gain in the
suite of automated tests.  The acceptance
tests actually define requirements in an objective and executable way.  If a tester finds a bug, he can write an
acceptance test that fails in the presence of the bug.  When the bug is fixed, the test will
pass.  Acceptance tests become an
important part of a regression test suite over time.

Stay tuned for more
detailed information on the following testing topics:

• Unit
  testing
• Integration
  testing
• Acceptance
  testing