Comments on: Code Coverage – Why? http://pathfindersoftware.com/2009/05/code-coverage-why/ The Fastest Way to Launch Successful Software Thu, 19 Jan 2012 16:36:03 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: sahaab http://pathfindersoftware.com/2009/05/code-coverage-why/#comment-9509 sahaab Sat, 04 Jul 2009 17:37:15 +0000 http://www.pathf.com/blogs/?p=2404#comment-9509 imho code coverage is a very misleading metric and i like to keep distance with it. it wouldn't cover most serious bugs and gives you a false feeling of security. you would also find lazy/bad developers who tend to write test cases with one goal in mind: 100% should happen and that will be it. well it might make sense in perl, where if nothing it tells you the syntax is alright :) heres what i would profess: 1. have automated regression test suite 2. for every feature you add have an exhaustive set of test cases added to the suite imho code coverage is a very misleading metric and i like to keep distance with it.
it wouldn’t cover most serious bugs and gives you a false feeling of security. you would also find lazy/bad developers who tend to write test cases with one goal in mind: 100% should happen and that will be it.
well it might make sense in perl, where if nothing it tells you the syntax is alright :)

heres what i would profess:
1. have automated regression test suite
2. for every feature you add have an exhaustive set of test cases added to the suite

]]> By: John McCaffrey http://pathfindersoftware.com/2009/05/code-coverage-why/#comment-9508 John McCaffrey Fri, 15 May 2009 14:09:50 +0000 http://www.pathf.com/blogs/?p=2404#comment-9508 My take on it is that if the code coverage goes down *significantly* by some agreed upon amount, that should be a build breakage. I think that point is going to be lost behind the 100% number. "Coverage Driven Testing" is a very real problem, and I've seen first-hand that people can get too caught up in the numbers, but as we know, the code coverage numbers don't mean that the tests are any good. Imagine the coverage was at a respectable 70%, and then someone checks in a big change that brings the coverage down to 60%, but the tests are all still passing. Wouldn't you want to know that your coverage just went down by 10%? If it turns out that the person found a bunch of useless tests and got rid of them, that's fine we'll reset our number. There are always exceptions to rules, but I just want to be Alerted to big changes. (this is even more true when working with bigger teams or people you don't know as well) If I was on this project with Jason, I would probably suggest that the build should break if the coverage went down by a certain percent, let's say 2-5% depending on how big the codebase is. As Noel pointed out in his post, there are two phases 1. Detect 2. Fix and I agree with him that its the Fixing part that is more important and indicative of how well your team works together. My take on it is that if the code coverage goes down *significantly* by some agreed upon amount, that should be a build breakage. I think that point is going to be lost behind the 100% number. “Coverage Driven Testing” is a very real problem, and I’ve seen first-hand that people can get too caught up in the numbers, but as we know, the code coverage numbers don’t mean that the tests are any good.

Imagine the coverage was at a respectable 70%, and then someone checks in a big change that brings the coverage down to 60%, but the tests are all still passing. Wouldn’t you want to know that your coverage just went down by 10%?

If it turns out that the person found a bunch of useless tests and got rid of them, that’s fine we’ll reset our number. There are always exceptions to rules, but I just want to be Alerted to big changes. (this is even more true when working with bigger teams or people you don’t know as well)

If I was on this project with Jason, I would probably suggest that the build should break if the coverage went down by a certain percent, let’s say 2-5% depending on how big the codebase is.

As Noel pointed out in his post, there are two phases
1. Detect
2. Fix
and I agree with him that its the Fixing part that is more important and indicative of how well your team works together.

]]> By: Daniel http://pathfindersoftware.com/2009/05/code-coverage-why/#comment-9507 Daniel Thu, 14 May 2009 20:04:23 +0000 http://www.pathf.com/blogs/?p=2404#comment-9507 One way I’m exploring right now to address this is to run coverage tests for controllers, helpers and models individually and exclude the other parts from the coverage statistics. This way controller specs don’t mess up helper or model coverage results. One way I’m exploring right now to address this is to run coverage tests for controllers, helpers and models individually and exclude the other parts from the coverage statistics. This way controller specs don’t mess up helper or model coverage results.

]]> By: Mike Heath http://pathfindersoftware.com/2009/05/code-coverage-why/#comment-9506 Mike Heath Wed, 13 May 2009 18:44:14 +0000 http://www.pathf.com/blogs/?p=2404#comment-9506 When you ask for 100% test coverage, you're asking for crappy tests. Code coverage is not a code quality metric. It's not even a test quality metric. Don't pretend that it is. When you ask for 100% test coverage, you’re asking for crappy tests. Code coverage is not a code quality metric. It’s not even a test quality metric. Don’t pretend that it is.

]]> By: Michael King http://pathfindersoftware.com/2009/05/code-coverage-why/#comment-9505 Michael King Wed, 13 May 2009 13:27:41 +0000 http://www.pathf.com/blogs/?p=2404#comment-9505 Don't you just love rcov's drive by coverage? While I think 100% coverage is a worthy goal, I think I am going to have to side with Noel in that it's not realistic. To mangle a quote, 'I do not think 100% coverage from rcov means what you think it means.' If rcov is improved to define coverage as assertions then 100% coverage means significantly more. Unfortunately that is easier said than done. Don’t you just love rcov’s drive by coverage?

While I think 100% coverage is a worthy goal, I think I am going to have to side with Noel in that it’s not realistic. To mangle a quote, ‘I do not think 100% coverage from rcov means what you think it means.’

If rcov is improved to define coverage as assertions then 100% coverage means significantly more. Unfortunately that is easier said than done.

]]>