A new version of TeamCity is out, and while I have not taken it for a ride yet, I found this feature particularly interesting:
Risk Group Tests, Tests Re-ordering
TeamCity 4.0 is now able to determine a set of tests which are likely to fail (i.e., recently modified, those with frequent failures history, etc.), and perform those tests first the next time the project builds. This allows teams to confirm potential fixes sooner, reducing feedback time.
There are strong cases to be made for risk-based testing in certain environments. The reason why it is not more commonplace is that implementation is tricky since it requires the CI system to know a few things which are hard to generalize:
- The type of build and testing framework
- Historical perspective across multiple builds
- A good set of heuristics along the way
Since a risk-based approach may mean skipping certain tests on some builds, this may sound like a step backwards to the agile developer. But consider it an alternative to improving overall build time (i.e. “based on very low risk assessment, tests which never failed will not be run every time.”). The hard part is defining what “low risk” means to your project, and how much value you would get reducing overall build time.
This is obviously not an issue on well-run projects with sub-minute builds, but if you were to be dropped into an existing project with, say, a large code base and a fifty-two minute build (I’ve seen it), this can become one way to perform triage on an otherwise untenable position. You don’t need a new CI system to implement risk-based testing, but having this alternative can’t hurt.
