Comments on: Help, My Test Is Failing! http://pathfindersoftware.com/2009/07/help-my-test-i-failing/ The Fastest Way to Launch Successful Software Thu, 19 Jan 2012 16:36:03 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: oelmekki http://pathfindersoftware.com/2009/07/help-my-test-i-failing/#comment-9710 oelmekki Sat, 01 Aug 2009 07:48:39 +0000 http://www.pathf.com/blogs/?p=3399#comment-9710 @David Hainlin: It's also avoidable when running autotest/autospec. You even don't have to commit, just to save a file and see your tests crash. -- "tests failing and app working" is something that regularly happens to me. Each time, i think "oh no, please tests, you're supposed to save my time". I think the biggest problem is the discordance of environments. It's especially true when doing integration testing with cucumber/webrat. We don't actually test our app, we test our test-app, which can have a really different environment from the production one. I have thought about this several times, without finding a perfect solution. Maybe something like using git to branch the app in a test branch, and duplicating dev db in test db without using fixtures or mocks for integration tests, testing more the value change between actions than the value itself... @David Hainlin:

It’s also avoidable when running autotest/autospec. You even don’t have to commit, just to save a file and see your tests crash.

–

“tests failing and app working” is something that regularly happens to me. Each time, i think “oh no, please tests, you’re supposed to save my time”.

I think the biggest problem is the discordance of environments. It’s especially true when doing integration testing with cucumber/webrat. We don’t actually test our app, we test our test-app, which can have a really different environment from the production one.

I have thought about this several times, without finding a perfect solution. Maybe something like using git to branch the app in a test branch, and duplicating dev db in test db without using fixtures or mocks for integration tests, testing more the value change between actions than the value itself…

]]> By: David Hainlin http://pathfindersoftware.com/2009/07/help-my-test-i-failing/#comment-9709 David Hainlin Fri, 31 Jul 2009 18:47:01 +0000 http://www.pathf.com/blogs/?p=3399#comment-9709 Nice post. This scenario is rarely encountered when using continuous integration - the offending commit is caught within seconds so you still have some fresh context. Running cruisecontrol for rails and java projects has all but remove this issue. On projects where there isn't a little robot friend checking all the commits, a tool like fisheye is fantastic for walking history and looking at the diffs. I used this quite a bit on a highly distributed team while cruise was down (being moved). Nice post. This scenario is rarely encountered when using continuous integration – the offending commit is caught within seconds so you still have some fresh context. Running cruisecontrol for rails and java projects has all but remove this issue.

On projects where there isn’t a little robot friend checking all the commits, a tool like fisheye is fantastic for walking history and looking at the diffs. I used this quite a bit on a highly distributed team while cruise was down (being moved).

]]> By: Ryan Bates http://pathfindersoftware.com/2009/07/help-my-test-i-failing/#comment-9708 Ryan Bates Fri, 31 Jul 2009 17:34:29 +0000 http://www.pathf.com/blogs/?p=3399#comment-9708 If a test is sometimes failing and other times passing depending on how it's run, it's likely due to some state left over from a previous test. To help debug this, try running your tests in reverse order. With rspec you can use the "-R" option. As a rule of thumb, every test should start with the same clean environment. Ensure the setup/teardown hooks are removing any changes leftover from a previous test. If a test is sometimes failing and other times passing depending on how it’s run, it’s likely due to some state left over from a previous test.

To help debug this, try running your tests in reverse order. With rspec you can use the “-R” option.

As a rule of thumb, every test should start with the same clean environment. Ensure the setup/teardown hooks are removing any changes leftover from a previous test.

]]> By: Dan DeLeo http://pathfindersoftware.com/2009/07/help-my-test-i-failing/#comment-9707 Dan DeLeo Fri, 31 Jul 2009 17:31:03 +0000 http://www.pathf.com/blogs/?p=3399#comment-9707 Wow, this brought back some bad memories from last night. On my Moqueue project (mock objects for the amqp library, it's a bear to test otherwise) I'm using the examples from the amqp library as integration/acceptance tests. One of these has a Logger class that demonstrates fanout queues, and it suddenly started failing, which confused the (perl code here) out of me. The culprit was wycats/bundler which was pulling "logger" in from stdlib when rubygems got required. The moral: be very thorough when thinking about what's changed since the test passed. Wow, this brought back some bad memories from last night. On my Moqueue project (mock objects for the amqp library, it’s a bear to test otherwise) I’m using the examples from the amqp library as integration/acceptance tests. One of these has a Logger class that demonstrates fanout queues, and it suddenly started failing, which confused the (perl code here) out of me. The culprit was wycats/bundler which was pulling “logger” in from stdlib when rubygems got required.

The moral: be very thorough when thinking about what’s changed since the test passed.

]]>