Reality
In reality, developers don’t like to do much testing. Developers aren’t testers. We typically will write code while making
certain assumptions about variables and, objectively knowing the expected behavior
of the syntax, we might run it once and call it done. Typically, bugs hide where the code wasn’t
rigorously tests or in paths that weren’t tested at all. I think everyone agrees that without testing,
the software will have bugs (and often even after testing). I heard of a crazy management quote that is
very sad: “If you have to test it, you aren’t building it right.” I sure am glad that manager wasn’t involved
with automobile development! In reality,
we need testing
Types of common
developer testing
- Actually
running the code written - Simple
console application or winforms test harnesses - Running
the application through the UI locally or on a development server.
Types of testing
- Unit
testing - Integration
testing - Acceptance
testing - Load
testing - Performance
testing - Security
testing - Exploratory
testing - And
many more
Approaches
- Do all
manual testing with or without the help of small tools. - With
this approach, the cost is high because for every run of the test, human
time is required. For every
release, every test case must be repeated. - This
approach doesn’t scale. When more
testing is needed, that directly calls for more human time. Human time is very expensive compared
to computing time. - Automate
all testing. - The
cost is low for about 80%, but then goes up. - Some
types of tests are hard or impossible to automate effectively, such as
security, exploratory, and concurrency testing. This type of testing needs more human
attention and isn’t easily repeatable. - To
fully succeed at this approach, testers need to be highly skilled at
scripting. - A
pragmatic approach – automate the testing that is easiest. - Unit
testing - Integration
testing - Acceptance
testing
Unit testing
What is a unit test?
From Wikipedia: A “. . . unit
test is a procedure used to validate that a particular module of source
code is working properly”. A unit test
is a very unique type of test with some fundamental constraints. If a test violates a constraint, it ceases
existence as a unit test.
Characteristics of a
unit test
- A unit
test isolates and tests a single responsibility.
- A unit
test isolates a failure. - A unit
test simulates other collaborators using various methods such as fake
objects. - A unit
test can run and pass in any environment.
When an automated test reaches out to a file, registry key,
database, web service, external configuration, environment variable, etc, it
becomes an integration test and is polluted with dependency on its
environment. At that point, the binary
can no longer be sent to another location with the expectation that test
continue to pass.
Unit testing is hard.
It requires constant vigilance to ensure application code is
loosely-coupled. Because a unit test
must test a piece of code in isolation, the code must be designed so that
collaborators can be easily separated.
In short, calling out to a static method from inside a constructor is a
tight-coupling. This type of code will
hinder unit testing.
Integration testing
What is integration testing?
An integration test is a developer test that combines several units of
code and tests the interaction among them.
A good integration strategy will include tests along all integration
boundaries. If a unit of code is tested
at the unit level and then tested during interaction with all collaborators,
the developer has confidence that the code will work well when the entire
system is assembled. Integration
testing
is the second step in a “test small, test medium, test large”
strategy. Larger integration tests are certainly useful, but
large tests alone don’t help to pinpoint a problem if they fail.
Characteristics of a
good integration test
- It
only includes a few classes. It
typically focuses on a single class and then includes collaborators. If a class has 3 collaborators, the test
would include 4 classes (the first class and its 3 collaborators). This test would aim to fake the
collaborators twice removed so that the focus on the test can remain
narrow. This is a subjective rule,
and the developer should use good judgment regarding how many classes to
include in an integration test. - It is
completely isolated in setup and teardown.
Should not be environmentally sensitive. If the test requires a file to be on the
drive, then setup for the test should place the required file there
first. If the test needs data in a
database, the setup should insert the required records first. It should not depend on any environment
setting being present before the test is run. It should run on any developer
workstation. - Must
run fast. If it’s slow, the build
time will suffer, and you will run fewer builds. - It
must be order-independent. If an
integration test causes a subsequent test to fail, it is indicative of a
test not owning its own environmental setup.
Acceptance testing
What is acceptance testing?
An evaluation of acceptance criteria on the system to ensure the system
meets the customers’ needs. Before a
piece of software can be written the software team must have acceptance
criteria. This is often called
requirements. Acceptance testing seeks
to execute the requirements on the system for a pass/fail result.
Characteristics of a
good acceptance test
- Easily
understood by non-developers as well as the customer. - Easily
created by non-developers as well as the customer. - When
it passes, the customer is assured that the system behaves as needed. - Very
expressive using jargon of the business domain. - Is
repeatable.
Automated acceptance testing is the final big gain in the
suite of automated tests. The acceptance
tests actually define requirements in an objective and executable way. If a tester finds a bug, he can write an
acceptance test that fails in the presence of the bug. When the bug is fixed, the test will
pass. Acceptance tests become an
important part of a regression test suite over time.
Stay tuned for more
detailed information on the following testing topics:
- Unit
testing - Integration
testing - Acceptance
testing