I was having a conversation with a co-worker this week about the automated build server on a particular project. My co-worker had set up the server to fail the build if test coverage dropped below 100%, and I was less than convinced that this was a good idea.
We didn’t debate it much, I wasn’t doing a very good job of making my points, and we both had other things to do, but it got me thinking about process, community, and incentives. So I thought I’d ramble on about that for a while.
Thinking about it, I decided that having the build fail on a coverage drop, although I still think it’s overkill, is not nearly as important as what happens after the build fails. Is it allowed to stay failed? Does the responsible developer fix it sheepishly? Is the responsible developer forced to wear a silly hat? What incentive is there for the developer to ensure the build status before checkin?I’m a fan of process, though I think that what I mean by process is generally more light-weight than what most people think of as process. I’m an even bigger fan of setting up technical and community environments so as to make doing the right thing easier than doing the wrong thing. My classic example is the original Apple Macintosh programming environments, which made it much easier to write a program that conformed to Apple’s Human Interface Guidelines than it was to stray from them. Developers tend toward the lazy, making it easier to match a guideline makes it more likely that it will be followed.
In this case, I might write or use a checkin script that runs unit tests and coverage tests before doing a checkin, and if it sees errors or a coverage drop, asks the developer whether they really want to go through with the checkin. This enforces the idea that test passing and coverage is important, while giving the developer the agency to override when necessary. Hmm, as I type that, I really like that idea. Gotta go…
Kidding. I’ll do that later. Actually, I’m sure somebody else has done that. Which reminds me of Flashbake. Flashbake is billed as really simple version control for writers or non-coders who have some basic terminal skils. It’s a simple front end to git that takes a set of files, and automatically commits them to git every fifteen minutes. It can be set to add useful information to the commit message, and it can even rendezvous with GitHub automatically. Again, the idea here is to make it as easy as possible to do the right thing — in this case, autosaving.
This is getting off the track, before I go to far off course, let me put it this way:
Telling people the right thing to do is a start.
Showing people the right thing to do is even better.
Setting up an environment and a community where the right thing is encouraged is the way to sustainably change your organization.
I’m a big fan of tracking the code coverage numbers, and failing the build if it drops significantly. The two scenarios I’m mostly interested in catching are #1 Deleted Tests: if someone goes through a refactoring or some other big change and deletes a bunch of tests, I want to know if that leads the coverage numbers going down (a true ‘refactoring’ should not lead to coverage going down, and any other major ‘overhaul’ that leads to lower coverage is something to watch out for)
The #2 scenario is if someone checks in new code that is not sufficiently covered.
In a way its encouraging Test Driven Development, and if the team is already working that way this is not an issue at all. But for those still not fully into TDD, this is at least encouraging ‘Test before Check-in’ Development, which is still a good thing.
If you establish these expectations up front, usually as part of your team Working Agreement, then everyone is on the same page, and its just how the team works. If you are adding this to a project already underway, it can be a little more challenging, particularly if it catches someone off-guard.
As an aside, I like to take the build breakage stats during the development phase and compare them to the number of bugs for those same features once they are out in production.