Every programmer tests, but does every tester program?  I submit that the good ones do. 

Jason Darling sent me these two great articles that dig deeper into this discussion.

http://www.testobsessed.com/2007/01/17/tester-developers-developer-testers/
http://chrismcmahonsblog.blogspot.com/2007/01/developer-testers-and-tester-developers.html

Recently, my team needed to produce an installer for our software package.  We wanted a neat setup package that installed prerequisites as well (.Net 2.0, and SQL Express 2005).  We decided to try out the Visual Studio 2005 setup project.  Here’s my experience report:

• Creation of the project was easy, but added files to include wasn’t so easy.  The setup project was default output of other projects as deployment artifacts.  This assumes a developer is building the GA release on a workstation.  My team uses a clean build server for all builds.
• Adding custom steps was nontrivial.  There were no hook points where I could pass in a command line to run.  I ended up creating a C# class derived from “Installer” just to Process.Start() a shell command.  Once I had that, I ran the shell commands necessary to set up the software.
• Creating a start menu shortcut was very difficult.  It appeared that it could create one for default project output, but again, I already had my GA release candidate.  I just needed to package it up for install.  I ended up creating another C# custom action to run Windows Script Object to create a .lnk shortcut file.
• Prerequisite installs worked like a breeze.  The installer detected the need for .Net 2.0, and Sql express and installed both seamlessly without unnecessary questions to the user.  Very pleased with this.
• The uninstall is good.  I have opposite steps for my custom actions, and the uninstall performs well.  Users get the standard experience of uninstalling from Control Panel.
• I have the option of including file a raw, loose file or bundled into a single .msi file. I opted for loose files to keep the install package flexible.
• PAIN POINT: I wasn’t able to automate the creation of the setup package.  The .vdproj file is not an MSBuild project file, and it requires Visual Studio just to build that project.  This means that I have to create the installer from a developer’s machine, not my build server.  This might be a deal-breaker for me, I haven’t decided yet.  It’s very important that everything be automated, and the only alternative is to install visual studio on the build machine (I’m not going to do that).

Overall, with a few caveats, the setup project is doing it’s job, and we’re shipping.  We’ll use it for the time being, but I’ll keep my eye on Nullsoft, Wise, and InstallShield.

Years of developer experience has produced a lesson the entire industry is paying attention to:  leaving integration to the end of a project is a bad idea!  It’s been proven a bad idea time and time again.  Integrating often keeps errors small and manageable.  In fact, many teams (mine included) integrate after every code check-in.  The idea of continuous integration is to make integration a normal part of your software construction process.  The core of continuous integration (abbreviated “[ci]”), is to feel the pain enough to make it go away.  The first pain is that it takes a LOT of time.  Solution:  automate the process.  We find that there are a lot of steps to integrate all the parts of a software system, so we automate all of them.  This yields a living artifact called a “build script”.  This build script is executable and runs every time someone commits a change to the codebase.  If a change to the codebase produces an integration defect, the build will fail, and the team will become aware of a new integration defect immediately (not weeks from now).  The build script helps refine the software process by identifying pain points and pushing the team to solve them.

A very popular [ci] combination of tools is [ccnet] for the build server (build reports, stats, and notifications), [nant] (an xml notation for the build script itself), [nunit] (for automated tests that run as part of the build), and the code compiler.  (Note, in Java, the parents of all these tools make up the popular combination: [cruisecontrol], [ant], [junit], and the java compiler).

With .Net 2.0, Microsoft has changed the way Visual Studio projects are structured so that they actually form an [msbuild] build file.  [msbuild] is Microsoft’s answer to [ant] and [nant].  The basic capabilities are the same (xml declarative build tasks, runnable from the command line).  [mstest] is Microsoft’s answer to [nunit].  Unfortunately, [mstest] isn’t available unless you purchase Microsoft’s Team Test IDE.  Microsoft doesn’t have an answer for [ccnet].

Before I go on, please take a look at Bil Simser’s experience report setting up CCNet with MSBuild and MSTest.

From reading Bil’s experience report, I am very glad that I saved my team A LOT of wasted time.  My team did not dabble with converting from [nant] to [msbuild] or from [nunit] to [mstest].  Here is my thought process:

• Our process works ([ccnet], [nant], [nunit], compiler).
• [ccnet] stays.  I don’t see an alternative that is compelling, and [ccnet] is not causing me pain.
• [nant] stays.  I see [msbuild] as a comparable alternative but with no compelling advantage over [nant] to force a conversion.  [nant] is not causing me pain.
• [nunit] stays.  I see [mstest] as a subset of NUnit’s functionality; therefore, there is no compelling reason to switch.  [nunit] is not causing me pain.
• compiler:  Switched to [msbuild] from the [nant] <solution/> task out of necessity.  The [nant] 2005 msbuild task is limited, and it caused me pain.  Moving to an <exec /> task calling msbuild.exe gave me exactly what I needed.  It’s been working for 6 months without further attention. 

To put my decisions in context, let me give you an overview of my department’s software process:

• Mix of [extremeprogramming], [scrum], and [crystal].
• 1-week iterations.
• Story wall of post-it index cards (story cards) for user stories as well as bugs.
• [ddd].
• OR mapping with [nhibernate]
• Model-view-controller Winforms UI
• Extensive automated test coverage.
• Automated configuration management.
• Automated database change and upgrades.

In this environment, we don’t have any time to waste.  We have to solve problems and move on.  We estimate in ideal hours, not days.  Anything that causes pain is either fixed or kicked out. 

———————————————

After all this, what does it have to do with Continuous Integration the Microsoft way?  I propose that this is the way to do it with .Net.  It works well, and teams all across the country do it in a very similar way.  From reading Bil’s experience report on attempting a mix of tools in the Team Suite stack, all he experienced was pain.  This isn’t the way to make tools.

To Microsoft:  you have to do better ( I know you can).  If by some odd change, these tools integrate well, and we just don’t know how to do it, please advertise the documentation that shows how to do it seamlessly in an automated, command line environment on a clean build server.  We all learn by example, so if teams within Microsoft are doing interesting things with [ci], please let us know because right now, we don’t know how to get it to work.

To the rest of the industry:  Do what works and evangelize it.  I’ve spelled out what is working well for my team, but there are plenty of other solutions to the [ci] problem.  If your team is doing CI with another mix of tools, please tell us about it!

My team has used Virtual Server 2005 and a single beefy physical server to do some great things this week, and I have to share.  The problem we are faced with is that we need multiple test environments for release and deployment testing.  We have a release very soon, and we’ve been doing our normal testing every day (automated testing, customer acceptance testing, etc), but now we are getting the whole company involved to lay a critical eye on everything.  Then we’ll package up the release candidate and do our deployment testing (Windows app- installed on the client).  We need to make sure the setup experience is great.

Situation:

• We need 6 test environments for release testing.  They should all have the software set up for testing.
• We need to clean these 6 environments multiple times to rerun tests.
• We don’t have 6 extra servers or client machines.

Solution:

We used Virtual Server 2005 to create a virtual pc with XPSP2 on it.  We then created a differencing disk and installed the software with the correct configuration for testing.  Then, we created 6 differencing disks that referenced the first.  We allocated 400MB of RAM to each 6 that need to run, and we have our 6 test environments.  The two root disks are there read-only since they are the base for the 6 live environments.  We enabled undo disks on each of the 6 test images, and now we can have all of our people run tests, and we scrap the undo disk to roll back changes to the clean, freshly set-up state.

When getting these set up, we did have one hiccup – we couldn’t access the virtual pcs from the network except through the ip address.  Solution:  Windows firewall was blocking network name access.  Fix:  “Allow incoming ICMP request”.  Yep.  Because we couldn’t ping the machine by name, we couldn’t remote desktop into it except by IP address.  We allowed ping, stopped, restarted Windows firewall, and it’s all gravy.  (In case you don’t know about this setting, check out http://www.cam.ac.uk/cs/docs/faq/m8.html).