Since the dawn of Extreme Programming, we’ve been told by some experts that agile development isn’t appropriate for safety-critical, highly regulated software, such as medical device and space flight software. Apparently a lot of teams didn’t get that memo. As you saw in Chapter 20, “Agile Testing for Mobile and Embedded Systems,” we’ve heard many success stories about teams in regulated domains such as aerospace and medical devices that have embraced agile values and principles. Many businesses must comply with financial regulations such as Sarbanes-Oxley (SOX). These companies have adapted agile development to fit their needs, and they’ve found how they can reduce risk with agile practices.
Some articles explaining why agile is not appropriate for regulated domains cite lack of documentation. We continue to hear this fundamental misunderstanding even as the term agile development is well into its second decade. As we’ve mentioned earlier, agile development is about delivering value frequently, at a sustainable pace. There’s no inherent reason that software developed using agile values, principles, and practices would have inadequate documentation.
When we use an example-guided approach to software development, we turn examples into tests, and we can automate many of those tests. As David Evans and Gojko Adzic have noted (Adzic, 2011), the automated tests become living documentation. The beauty of documenting code by way of automated tests is that since you have to keep regression tests passing, you are always keeping your documentation up-to-date. We gave an example of this in Agile Testing (p. 402). Accurate documentation is critical in domains where people’s lives, security, or money is at stake.
Regulatory agencies impose rules, and everyone governed by those rules must prove that they are complying with them. We must test to make sure that our software that tests compliance produces accurate results.
Lisa’s Story
In my eight-plus years working on a team that developed a financial services application, I got lots of experience with verifying that our software complied with federal regulations—or that it helped our customers comply with those regulations.
Our customers used our software to manage employees’ 401(k) retirement plans. Each year, they had to prove that they operated within IRS rules. For example, executives couldn’t take advantage to pad their own accounts with tax-free savings. The rules were extremely complicated and often counterintuitive, but I enjoy learning challenging business domains.
Our product owner gave us many examples of sample inputs and expected outputs. Turning those into tests that guided coding ensured our success. We fearlessly delivered highly complex software that reliably automated compliance testing for our customers, so they weren’t at risk for incurring fines.
Our competitors couldn’t believe we were able to do this. But we had the tests to prove it!
Later, we were faced with “Bernie Madoff rules,” regulations to help prevent brokers from scamming investors. These involved things like disclosing record-keeping fees and preventing plan advisers from changing investors’ personal information. The software to comply with these rules didn’t generate revenue, but failure to comply could mean significant fines for our company and our customers. For each regulatory requirement, we worked with the business stakeholders to find the simplest software solution, minimizing cost but ensuring compliance and security.
Audits and regulatory submissions impose accountability and other requirements needed to get a stamp of approval. Many of those regulations say, “show evidence that” or something similar; unfortunately many organizations often interpret this to mean, “give me more documentation,” and they spend their time, not testing, but documenting (see Figure 21-1).
Janet’s Story
A while ago, I was at a company that wanted to start doing more agile development and still needed to be SOX compliant. I looked at the company’s standards and saw that they were anything but agile. The test strategy document had layer upon layer of required documentation, and the people who were tasked with writing it weren’t even sure who was reading it, or why it was needed.
The agile teams were struggling to figure out what was absolutely necessary to fulfill their compliance but not get overwhelmed in documentation that had little meaning and did not satisfy their needs.
There are many companies that are able to comply with regulatory rules without excessive documentation (see Figure 21-2). Generally, as long as the company documents what it does, does what it documents, and can produce evidence that it is doing what it says it is doing, regulators will be fine with that. It is worth your while to read the regulations and then take a pragmatic approach to meeting them. For example, taking photos of the story board on a regular basis and having those available for review may be a perfectly legitimate way to document what has happened for compliance purposes.
Make your auditors (they are stakeholders) part of the solution, and define their needs as stories or tasks so the deliverables are built into the solution. Consider tasks that need to be completed to comply with regulations when your team creates a definition of “done” for the next release. Teams need discipline to follow their process consistently and produce the required artifacts. Testing activities contribute much to ensure compliance. As mentioned earlier in this chapter, automated tests are a great way to create consistent test results and living documentation. Compliance is part of Release Done, and therefore part of the product, not an exam to pass at the end.
Janet gets asked one question fairly often: “Are there any domains that shouldn’t attempt to use agile methods, like regulatory?”
What a great message—make your regulatory requirements part of your work, constantly thinking about how to make them as simple and effective as possible.
Testing in regulated environments is an opportunity to grow our T-shaped skills by collaborating with programmers and other team members, as you will see in the next story.
JeanAnn’s team found a lightweight solution for traceability of requirements that was acceptable to their regulatory auditors. There’s a perception that regulated always means heavyweight, but in our experience, auditors are usually open to simple alternatives to meet their information requirements. Experiment and collaborate to find an optimal solution.
Many aspects of agile development, such as short feedback loops and automated regression tests to provide living documentation, make agile practices suitable, even advantageous, in regulated environments. Bringing the subject matter experts together with testers and other development team members allows us to guide coding and mitigate risk with a wide range of examples. Regulated environments provide unique opportunities for testers who enjoy mastering business rules and helping the team deliver the right thing. Some of the aspects of agile approaches to testing in regulatory environments we covered in this chapter are:
Contrary to popular belief, agile does not mean no documentation. Teams that use examples to guide development turn those into automated tests that provide living documentation that can help meet regulatory requirements.
Complying with auditors’ requirements for information doesn’t necessarily mean producing old-school, heavyweight documents. Work with auditors to find lightweight ways to comply that fit with your team’s needs.
Government regulations add quality characteristics that need to be verified, even if they don’t directly add business value. Avoiding fines and ensuring customer security are critical to the business.
Tester-programmer collaboration can be essential for testing high-risk, regulated software products. Understanding the system design, hardware considerations, and timing issues helps in preventing defects or finding them early.
