Archives

Very few people in the software development community have issues with maintaining good attention to the details. However, I bet those who live in the regulated software community view the “normal” software world as quite sloppy. Attention to detail is a matter of life and death for a medical device. Because of this, the entire software community does a good job maintaining checks and balances. I have three suggestions that will make a tremendous difference when you start a regulated project or when you finally get near to releasing your medical device.

Photo by: myguerrilla

#1 Plan to get audited

It is a good bet you are going to get audited sooner or later. Its even possible to get a product out into the market only to have it get audited a second time. An FDA auditor will keep asking “why do you think that works?” type questions and continue to drill down until they find something or are satisfied there is nothing to find. If they find something you are going to get a warning letter and start the expensive process of FDA remediation. Getting past this will cost you a significant amount of resources and likely delay new product development. Usually just in time for your competitors to catch up. Keep good records.

#2 Do Test Driven Development (TDD) Well

By well I mean do all of the normal activities that are part of TDD … and do one more thing. Create self documenting code that also lends itself to traceability directly back to product requirements. Imagine telling an auditor that you can prove that you tested every path of the code EVERY time you checked your code in…thousands of times.  All of these tests can be directly linked back to product requirements. That would make an audit pretty easy.

#3 Create a Traceability Matrix

This may seem obvious. However, I know of many, successful, software shops that have stopped doing this. Ask a developer with 1-2 years of experience why they would need a traceability matrix and you will know what I mean. If you are planing to get audited and want to show that all of your requirements have been tested thousands of times, you need a traceability matrix.

Next Up: Partnering

Testing is a crucial part of all software development. If you don’t believe this you are likely just starting your career or haven’t had to support an application in production with users.  It becomes doubly important in regulated environments like the ones for medical devices. This is because testing for regulated software serves multiple purposes. There is the usual software product development reasons to consider (i.e. professional integrity and no one wants a bug in production). But it is also because the FDA, in particular, wants to make sure you are performing validation and verification of the software. Verification (Was the product built right?) Validation (Was the right product built?)

While I acknowledge there are a number of useful and important ways to describe software testing techniques, I strive to keep things simple, and understandable, by the whole team. To that end, I split testing into two tracks: Functional and Technical. Functional is about bring business value to the stakeholders – the validation part. On an Agile project we are testing the acceptance criteria of each story. This validates we have realized the benefit to the business and end users. If some thing cannot be tested by an end user, it doesn’t exist or have significant value. This is of course untrue. However, to simplify things and keep the team laser focused on only working on tasks that provide business benefit I start with this ideal. All other testing activities are testing Technical requirements. Or, verifying that the software is correct. Some examples of these technical requirements are things like performance criteria, common libraries to the backend and data retention rules. Those who do testing for a living are looking at the screen sideways by now I suspect. I realize that these things must be done correctly or the system will not realize business value to an end user. Both ate needed. That’s why the FDA requires proof of verification and validation.

The common flow of testing in an Agile project is to create acceptance criteria for each story then do regression testing after each iteration:

On the other hand, tradition testing documents and verifies each step before it is performed. The goal is for planning to be exhaustive and cover all possible scenarios.

Traditional testing is expensive and often makes validation hard to do. On the other hand, the common Agile testing approach is going to make documenting verification activities difficult. Once you acknowledge both are needed you can find a way that works. The best tools we’ve found leverage and extend the automated testing you are already doing. Look at the software testing as having two “tracks”.

#1 Manual Functional Testing of acceptance criteria. All of the speed and simplicity of user stories. An efficient communication mechanism for stakeholders.

#2 Test everything else via automated tools. Ensures all gets tested. In our case we use unit testing and 100% code coverage.

Here is a graphic of how we can document that we are testing against pre-approved requirements – both functional (validation) and technical (verification).

Next up: Its not what you wear. Its how you wear it.

How many time have we heard some form of the phrase “it just a one line change”? Or “it shouldn’t effect anything else”?

I think we know the all to frequent result…..”I can’t believe THAT broke……!!”.

Good developers write automated tests to help surface side-effects like this before they get into production and cause real damage.

Good developer refactor code mercilessly to make modifications less error prone.

Good developers take the perspective that maintenance begin as soon as you write “just one” line of code.

Getting a version 1 product out the door is hard work. Keeping the cost of ownership reasonable afterwards, so others can maintain your code, requires a perspective of continuous maintenance.

edge case city: requirements and testing dates for HR business logic

We have an internal application that does staffing, time entry, and now Paid Time Off (PTO) accrual, scheduling and management. It is quite nice, as it has replaced three existing systems, and replaced a number of manual, tedious tasks. I started it last year, as our current system was very inefficient. It was a simple Ruby on Rails app that I was able to get working in a few weeks. Over time the functionality grew.

We recently added in PTO accrual and functionality to debit PTO time. In doing so in a test driven manner was great, as we could put all the edge cases. What we found was that many of the requirements were plentiful with edge cases. For instance, we get paid on the 15th or end of the month, unless that is a weekend. Then we get paid on a Friday – Except if that Friday is a holiday, then the previous Thursday – Except – if that is also a holiday…. So it is really the previous non-weekend, non-holiday on or before the 15th or end of the month. The same holds true to determine the beginning of the billing period. If the Monday is the 2nd, than for some operations the effective billing period start is Tuesday the third…. Ugh…

Our initial thoughts were filled with visions of lots of very similar tests for all contexts that do things depending on the start or end of a billing period. We didn’t want to have lots of duplicate test code to test all edge cases, so we decided to extend the Date object to have a few helper methods to do the complicated logic in one place. We get the full range of the billing period start and end, and then trim off any holidays or weekends. What this did, was allow us to test the complicated logic in unit tests with a full set of complicated edge cases. We created many crazy examples of billing periods starting and ending on weekends, holidays, and holidays before and after weekends. This created a very robust set of methods to be used anywhere in the system.

We then mocked a call to each Date helper method everywhere in the system where it cared about what day it was, and weather it was the start/end of a billing period. We had only a few edge cases now: is it the true effective billing period start/end, or not. Our tests could then focus on the guts of what it actually did, regardless of the effective date.

This demonstrates how powerful unit testing is, and how mocking can really keep your tests concise. It made not only our code cleaner, but our tests cleaner. It reduced duplicate code, and increased our assurance that the code will function as designed.

That being said, it still doesn’t change the fact that implementing HR logic in any language is a pain in the ass. I have worked on a couple of systems for accounting departments in the past, and their business requirements are much worse than HR. Dealing with job codes, general ledger accounts, etc can make your head spin. All I know, is that without TDD, this system would be buggy.

Ask Mr. Lizard, from Jim Henson’s Dinosaurs

It’s time to play “Ask A Tester Person”, where I answer questions that I’ve gotten via email or otherwise about Rails Testing topics.

If you have a question for Ask A Tester Person, send it to railsprescriptions at gmail.com.

I’ve got two questions today:

Rails Test Prescriptions, the eBook put out by Noel Rappin, Director of Rails Development at Pathfinder, has been picked up by Pragmatic.

Congratulations to Noel – he’s done a great job of furthering testing best practices in rails, and this is a great reward. As he said “I’m very excited by this. I’ve wanted to work with Pragmatic for as long as they’ve been publishing books, and I’m thrilled that this particular project will be able to get wider distribution and access to Pragmatic’s editorial expertise and skill.”

* The current free “Getting Started with Rails Testing” ebook will continue to be available. If, at some time in the future, there’s a better Getting Started tutorial in the Pragmatic book, it may be offered as a replacement.

* The update site for current Rails Test Prescription owners will continue to be available for the foreseeable future.

* There will be one more official update to the current Rails Test Prescriptions, probably around the end of August. This will wrap up the chapter or two I’m working on, and tie up some other loose ends.

* After that, errata and information about changes to test tools will most likely be handled via this blog and an errata page on the rails test prescriptions site.

This is Noel’s 4th book with a major publisher, following Professional Ruby on Rails, wxPython in Action and Jython Essentials. We’re happy for Noel and happy to have him at Pathfinder.

Related Services: Ruby on Rails Development, Custom Software Development

Here’s a minor thing that bugs me all the time.

I’m writing a functional test:

should "do something functional"
  get :search,  rder_id => @order.id, :user_id => @user.id
  # and so on
end

The get call in that test simulates a browser request. Intuitively, you would (well, I would) expect this request to be identical to a request coming from the actual view, via a helper like link_to("search", :action => :search, rder_id => @order.id, :user_id => @user.id). At least, you’d expect that parameters hash in the controller to be the same between the

Makes sense, right? The testing call should set up the same environment as the actual call being tested.

Frustration, the game, photo by unlovalblesteve

Dot, dot, dot, dot, dot — tests are passing, looks like it’s time for lunch — dot, dot, dot, dot, F. F? F? But the code works. I know it does. I think it does. Why is my test failing?

One of the most frustrating times as a TDD developer is that moment when a test is failing and you don’t know why, as opposed to the more normal case where the test fails as expected. Here’s a grab bag of tips, tricks, hints, and thoughts to get us all through that difficult time.

Spam Wall, by freezelight

What with upward of two people saying nice things about last week’s post, I’ve decided to keep going with part two of a look at some real testing code.

Most code-heavy tutorials show the code but not the tests — I’m doing the opposite here, and showing the tests, but not much of the code. Also, although I’m presenting these tests in chunks, you should realize that there was a lot of back-and-forth from Cucumber to tests to code and some backtracking, most of which I’ll spare you from having to wade through.

At the end of last week, I had run through the tests for spam-prevention code which worked by limiting the rate at which a user could send messages to other users of a particular social networking site. Cucumber was involved, and I think I went off on a tangent about writing lots of tests.

Spam Wall, by freezelight

As sort-of promised in last week’s post, I’m going to work through a real-world test example, with an eye toward explaining how and why I tested the way I did. Hopefully, I’ll be able to do this at blog-post length. If not, well, there’s always next week.

This site, which was a legacy rescue, allows users to send messages to each other within the site without having to give away their other contact information. The problem is that nefarious spammer types were creating logins and immediately sending messages to large numbers of the user population, irritating them. After some deliberation, the client decided on a rate-limiting strategy, where a member could only send a certain number of messages in a day, and a new member could send even fewer messages a day. Messages above that point would require administrative action to unblock the user’s privileges.