Archives

Bob Moll, Pathfinder Software’s Lead User Experience Designer, and I gave a talk on Lean User Experience at the Chicago Lean Startup Circle on June 30th. User Experience is critical to the success of a software product: Well designed apps get used and recommended, poorly designed ones get discarded. But traditional user experience is slow and expensive and doesn’t get high quality products to market faster. This is death to any new product development effort. How can you get user experience at startup speed?

Enter Lean UX …

This may be generic advice, but it might not be a familiar concept for those in IT working within a large organization doing validated software:

Once you adopt the practices described in my previous blogs about regulated software, you will find that you aren’t working alone anymore. The tight feedback cycles allow you to regularly check-in with the business and involve them in building the software. Once this happens, priorities and features will change. Early on they will change often. Don’t discourage this as its a good sign things are moving in the right direction. The people talking to your customers are directly involved in building features your customers will use. This is exactly what you want because it increases the likelihood that you will produce a better product. Now that you have left your IT bubble you must partner with your stakeholders rather than simply delivering to them on a regular basis. How do you effectively partner with business and marketing? Here are the difficulties you will likely face:

• Goals are different
• The phases and corporate dialect are different
• The time frames and processes for decision making are different
• Required documentation will be different and address different goals
• Stakeholders will feel comfortable giving verbal requirements, but you must document them, get them formally approved and finally build them into working software

How do you partner with the business and still meet your commitments? Follow the practices we presented earlier and stick to the following principles:

• Consistent focus on activities that add business value
• Build trusted partnerships
• Constant customer engagement
• Continual process and practice improvement
• Be collaborative and transparent
• Work together: No heroes, lone wolves, or cowboys
• Pay attention to the team and adapt as needed

Very few people in the software development community have issues with maintaining good attention to the details. However, I bet those who live in the regulated software community view the “normal” software world as quite sloppy. Attention to detail is a matter of life and death for a medical device. Because of this, the entire software community does a good job maintaining checks and balances. I have three suggestions that will make a tremendous difference when you start a regulated project or when you finally get near to releasing your medical device.

Photo by: myguerrilla

#1 Plan to get audited

It is a good bet you are going to get audited sooner or later. Its even possible to get a product out into the market only to have it get audited a second time. An FDA auditor will keep asking “why do you think that works?” type questions and continue to drill down until they find something or are satisfied there is nothing to find. If they find something you are going to get a warning letter and start the expensive process of FDA remediation. Getting past this will cost you a significant amount of resources and likely delay new product development. Usually just in time for your competitors to catch up. Keep good records.

#2 Do Test Driven Development (TDD) Well

By well I mean do all of the normal activities that are part of TDD … and do one more thing. Create self documenting code that also lends itself to traceability directly back to product requirements. Imagine telling an auditor that you can prove that you tested every path of the code EVERY time you checked your code in…thousands of times.  All of these tests can be directly linked back to product requirements. That would make an audit pretty easy.

#3 Create a Traceability Matrix

This may seem obvious. However, I know of many, successful, software shops that have stopped doing this. Ask a developer with 1-2 years of experience why they would need a traceability matrix and you will know what I mean. If you are planing to get audited and want to show that all of your requirements have been tested thousands of times, you need a traceability matrix.

Next Up: Partnering

Its not what you wear. Its how you wear it.

Photo by country_boy_shane

I have to thank Tavi Scandiff-Pirvu for coining this expression as it relates to regulated software development. As long as you use industry standard methods, the FDA allows you to determine the rules that you are going to follow. If you want to have 15 exhaustive steps in order to build each piece of functionality the FDA will say, “Go do it. Just document each step as you go.” If you can reach the same quality in only 7 steps the FDA will say, “Go do it. Just document each step as you go. The FDA isn’t going to test your software in detail. They are going to test your process to ensure that you did what you said you were going to do. This may appear that the FDA is interested in the product but only the process! But, the FDA also wants to see (via documentation) that you built the product features you claimed you had.

The criminally minded will see an opportunity to claim they did more than they actually did via documentation, but in reality do less work. However, there is a natural equilibrium via the market. If you cut steps you will likely pay for it in your product via poor quality and sales. In the end the product has to work. Also, experience shows that it generally takes just as much time to design, create the documentation, necessary traceability and tracking support as to do the actual work. Inefficient? Slightly, but we are trying to sustain human life – not make a quick buck. Its not just about the money. Finally, a big picture analysis shows that it is cheaper for everyone to document what you did rather than pay somebody from the FDA to be a part of your project on a daily basis.

Don’t cut corners, or change your process, once you define it just to save time and money!

My final thought is that in order to do good process design (the 7 step version vs the 15 step version) and thorough documentation you will need to know your organization and team well. You will also need to know the product well so that you don’t design a seemingly efficient process that is blind to the needs of the market the product will be launched in.

Sorry kids. Nobody ever said it was going to be easy.

Testing is a crucial part of all software development. If you don’t believe this you are likely just starting your career or haven’t had to support an application in production with users.  It becomes doubly important in regulated environments like the ones for medical devices. This is because testing for regulated software serves multiple purposes. There is the usual software product development reasons to consider (i.e. professional integrity and no one wants a bug in production). But it is also because the FDA, in particular, wants to make sure you are performing validation and verification of the software. Verification (Was the product built right?) Validation (Was the right product built?)

While I acknowledge there are a number of useful and important ways to describe software testing techniques, I strive to keep things simple, and understandable, by the whole team. To that end, I split testing into two tracks: Functional and Technical. Functional is about bring business value to the stakeholders – the validation part. On an Agile project we are testing the acceptance criteria of each story. This validates we have realized the benefit to the business and end users. If some thing cannot be tested by an end user, it doesn’t exist or have significant value. This is of course untrue. However, to simplify things and keep the team laser focused on only working on tasks that provide business benefit I start with this ideal. All other testing activities are testing Technical requirements. Or, verifying that the software is correct. Some examples of these technical requirements are things like performance criteria, common libraries to the backend and data retention rules. Those who do testing for a living are looking at the screen sideways by now I suspect. I realize that these things must be done correctly or the system will not realize business value to an end user. Both ate needed. That’s why the FDA requires proof of verification and validation.

The common flow of testing in an Agile project is to create acceptance criteria for each story then do regression testing after each iteration:

On the other hand, tradition testing documents and verifies each step before it is performed. The goal is for planning to be exhaustive and cover all possible scenarios.

Traditional testing is expensive and often makes validation hard to do. On the other hand, the common Agile testing approach is going to make documenting verification activities difficult. Once you acknowledge both are needed you can find a way that works. The best tools we’ve found leverage and extend the automated testing you are already doing. Look at the software testing as having two “tracks”.

#1 Manual Functional Testing of acceptance criteria. All of the speed and simplicity of user stories. An efficient communication mechanism for stakeholders.

#2 Test everything else via automated tools. Ensures all gets tested. In our case we use unit testing and 100% code coverage.

Here is a graphic of how we can document that we are testing against pre-approved requirements – both functional (validation) and technical (verification).

Next up: Its not what you wear. Its how you wear it.

You started a new job recently. Your organization has a some “legacy” apps, plus they’re working on a new batch using some cutting-edge technology. You’re assigned to a cool team with some hot-shot developers and ready do get down to work.

Suddenly, on day two, your manager walks by to let you know that the lead (and only developer) on one of the apps has resigned and will be leaving next week. Your manager asks you to pick up work on the project.

“Well gosh”, you think. “What a great opportunity to show my stuff”.

After a brief discussion with your soon-to-be-ex colleague, you realize this might not be a picnic afterall. The system seems to be functioning well and has a hefty backlog of pending work. However, when you inquire as to any form of documentation, you get a blank stare in return. Not only that, but the unit tests have been breaking of late, so most of them have been commented out. The build is manual and the deployment scripts work most of the time.

Well….now what?

Two suggestions come to mind:

1) Get in a room with your colleague and start a simple set of diagrams which depict the system top-down. These should be high level to begin with; a few architecture diagrams covering the technical and the conceptual aspects. Next ask about the complicated parts of the system and get some interaction diagram out on the white board. Next, make sure all the build and deployment information is documented and that they actually work (you can reproduce those steps yourself). Finally, ask about technical debt. Create chores to fix all the “we don’t have time to fix that now” gotchas. Lastly, take pictures of everything and write up the info in simple form. Posting on a wiki is helpful and if you’re using a tracking tool like Pivotal or ScrumWorks, enter everything in there.

2) Talk to your manager about introducing pair programming into the team to avoid this kind of situation in the future. Make sure you keep the unit tests passing 100% of the time going forward.

The moral of the story: while system documentation is important, it can be created just-in-time. A good practice is to use a wiki to keep all environment information logged and up to date. Trying to keep system document current is unnecessary as no one will read it until……at least he/she didn’t get hit by a bus.

Lets be honest: Checklists aren’t the most exciting topic to blog about. However, in the context of regulation they become interesting. Everybody loves checklists and hates them at the same time. It’s really easy to make them and just as easy to forget to use them. Later on, after the emergency and/or reason they were created has subsided, using the checklists becomes a ceremony – something that must be done rather than a helpful tool. Checklist also may foster an anti-change mentality and allow people to cruise through their jobs without think about them. This is why we all hate them.

Photo By: adesigna

All feelings about checklist aside: If each detail of your project’s history is going to be scrutinized, checklists are essential. On a regulated project their are a lot of steps to follow. It’s hard to remember each step of each procedure followed. Its really easy to finish delivering a feature then forget to check the software verification report in to your document management system. Once this occurs it looks like you didn’t test the feature before checking it in. Worse, you moved on to the next step of features and never turned in the verification report ever. 9 months later an auditor finds it and you have to doubly prove you did the testing and that you didn’t miss any other steps someplace else. Nobody wants to be in a position where they are on the defense. Often a missed step means all of the testing and tracking is nullified.

There are number of things you can do to ensure you have good lists and they get used often.

• Make them short – A 10 page checklist is not going to be used properly or use up time at the cost of efficiency. Train your staff. Don’t rely on process to get quality work.
• Have less steps – Can you tighten up your process and simplify it?
• Create checklist by role rather than at stage gates – The list will be shorter and highly focused on what matters to the person doing the work.
• Have the team members create their own checklists – They can identify the handful of tasks that are at risk of getting missed. That’s your checklist!
• Review checklists often for new procedures and question why each step exists often.

Finally, keep in mind that whatever you put on the checklist becomes part of your project documentaiton history. Thus, you must do every item on the list every time you run through a procedure. Think carefuly before adding things to a checklist.

Next up: Functional Testing vs Technical Testing

How many time have we heard some form of the phrase “it just a one line change”? Or “it shouldn’t effect anything else”?

I think we know the all to frequent result…..”I can’t believe THAT broke……!!”.

Good developers write automated tests to help surface side-effects like this before they get into production and cause real damage.

Good developer refactor code mercilessly to make modifications less error prone.

Good developers take the perspective that maintenance begin as soon as you write “just one” line of code.

Getting a version 1 product out the door is hard work. Keeping the cost of ownership reasonable afterwards, so others can maintain your code, requires a perspective of continuous maintenance.

The FDA states that a medical device is a product that is used for medical purposes in patients, in diagnosis, therapy, or surgery. As opposed to a pharmaceuticals, who use the body’s metabolism, medical devices act mechanically or chemically. The FDA’s role in this is to assure the safety and effectiveness of devices. Keep this in mind while going through the process (they are not just to make your life hard).

Photo By: birlewphotography

From a software developer’s perspective, there are three classes of medical devices the FDA regulates. You should know what kind of product/device you are making very early on – before you start planning the work. Their are legends of companies that have outsourced development only to find that their partner had yet to take a device to the point where the FDA would audit it. An even worse scenario: they had done the work with an offshore partner who didn’t fully understand FDA requirements and couldn’t prove traceability nor pass an audit. Whatever cost effectiveness a company offers, make sure its not going to be at the expense of doing the project twice.

Class I

Class I devices are not intended to be used for supporting life. As such, they have the least regulatory controls. Product development efforts lend themselves well to utilizing Agile practices. Do some research before you start development though. About 74% of devices are exempt from pre-market notification in the first place. Those that aren’t, might fit into an existing device panel. Often a few pages of documentation will suffice to pass pre-market approval for Class I devices. To find out more go to the FDA’s classification database.

Class II

For Class II devices, the FDA recognizes that general commercial quality control and manufacturing practices alone may not be sufficient to assure safety. However, they state that “existing methods” are in place to prove safety. You will still have to prove this to the FDA. Simply put: If you follow a process, and document it, you will be able to prove the device you are creating is safe. Rely heavily on documenting the history of the project well. The project will be characterized by significant documentation and process compared to a “normal” Agile project. Because Agile is quite new in the FDA space, you will probably need a seasoned compliance officer with FDA experience. Plan to get audited. This doesn’t mean that utilizing Agile practices is difficult or shouldn’t be done. In fact, because so much time is spent on each feature of the software, I would argue that efficiency and managing waste become critical issues. Attacking risk and focus on delivering the highest value features first is key here. The team just can’t be sloppy with process or documentation for this type of device.

Class III

Class III devices are new or are used to sustain life. The team can’t rely on existing methods to prove to the FDA that the device is safe and labeled correctly. Time to market will play a part, but software development isn’t likely to be on the critical path of product development. The raw science and research are likely to take up most of the product lifecycle. Is agile right for R&D? That’s a topic better covered in another blog post.

Given the wide range the FDA has in place, the life of your product development team will be greatly affected by the classification of the device you are creating. In my experience, working on Class I and Class II devices often feel like working on a “normal” Agile project – assuming your Agile process incorporates enough controls to document the process itself.

Next Up: Keeping Track with Checklists

Methodology

“The analysis of the principles of methods, rules, and postulates employed by a discipline”

For software companies these day it seems that its not a hard choice to follow Agile principles. With some practice, or hiring a consultant, you can learn Agile practices and employ them as well. Its an entirely different animal to decide you are going to use Agile practices in a regulated environment though. If you choose this route you must understand that you are on the leading edge of the rest of the industry. Here are some questions you should know the answer to before you can get started:

Photo By: susanrudat

What is Right for My Organization?

Most companies operating in a regulated environment are comfortable utilizing traditional software practices. This is often a waterfall approach. However, it may not be. Most Agile practices are not new – they are what good software organizations have been doing for years. Before you decree “We will be Agile” take a look at what the software organization is doing now. Do an inventory of the practices they are doing compared to something like Kanban or Extreme Programming. Before changing the process for process sake, ask yourself: Are they meeting deadlines set? Are you happy with the products they are creating? Does the overall organization seem happy with the software group?

If the software group isn’t delivering then using Agile practices can be a proven tool one can use to solve the problem. That being said, some organizations are extremely resistant to change. Being transparent, working collaboratively, and focuses on delivering software regularly can hard for somebody who has spent their career doing the exact opposite. In this case Agile may not something everyone can embrace easily.

How fast can the organization move?

Eventually a team that embraces Agile principles and practices is going to start moving faster than before. Make sure the rest of the organization understands clearly what is going to be expected of them. This will require open communication and and transparency by your team. There is no need to have a rockstar team that can’t move forward because the backend team isn’t able to deliver dependent software within your schedule.

Do I know enough about Agile?

My experience is that the most popular Agile templates out there (Scrum, Extreme Programing, Kanban, Lean) don’t directly lend themselves to building software in a regulated environment. This means are you going to be bending and breaking the “Agile” rules a bit. Before you can make your own path you must fully understand the “normal” path. Make sure you have significant Agile experience or hire that experience in before you start something difficult and try to expand on it.

Next Up: Have Some Class