Quality Assurance for the new millennium

“The quality goes in before the name goes on”–Zenith Electronics slogan.

“Quality is job one”–Ford Motor Company slogan.

When I worked on an automotive assembly line in the early 1970s, they had a Quality Control department. It acted like a sort of quality high-pass filter: incoming parts and assembled trucks were inspected; if they were high enough quality, they were used or shipped; if not, they were scrapped or sent to the rework department.

Over the years, Quality Control departments transformed themselves into Quality Assurance departments, the idea being that rather than just filtering out low-quality products, they would endeavor to make sure the products got built right in the first place.

The Toyota Production System refined this idea with poka-yoke (error proofing) and the andon light (where anyone on the line could signal a problem, and the line would stop while they figured out how to keep the problem from happening again).

Sadly, many software development organizations, particularly in mature companies, are still stuck in the era of Quality Control. The QA department finds bugs and reports them, the bugs are fixed, and charts are distributed to management so they know which groups to beat up about having too many bugs.

What is missing is the big picture of quality: it is not a department that acts as a gate, or even worse, gets rated on how many bugs they find. (I once worked with a QA department that sent very vague bug reports to development, things like “This message may be wrong.” Their explanation was that they got rated on how many bugs they found, so their boss didn’t want them to waste any time doing research on whether or not something really was a bug.)

Since we’re talking advertising slogans, remember the Maytag repairman, who never had any work because Maytag washing machines were so good? That’s kind of what we are shooting for, but we don’t have a QA department that is sitting idle. Instead, we have a QA presence on each Scrum team (as required in order to have cross-functional teams), and in addition to figuring out how best to test things, he or she also helps the team design for quality and for testing.

Poka-yoke applies to software in many ways. Modules should be small, and should adhere to the Single-Responsibility Principle. (Robert C. Martin: “Functions should do one thing. They should do it well. They should do it only.) Their APIs should be well documented, preferably in extractable comments, like Doxygen or Javadoc, so they are more likely to stay in sync with the code. Their actions should make sense; in other words, when a programmer makes an assumption about what the module is going to do, he or she should usually be right.

Program source tends to get sloppy over time, particularly when you have new programmers who are accustomed to different bracketing styles. This can cause errors when a bracket isn’t where it is expected and is overlooked. (Keeping modules small makes this a lot less of a problem, though.) Program source can be standardized to a common bracketing style using tools like Artistic Style.

Following the Single-Responsibility Principle makes it easier to unit-test modules. If a bunch small modules that have been thoroughly unit tested are assembled to make a product, the amount of testing required for the complete product is reduced. We still have to test to make sure we didn’t have misunderstandings in the design or assembly, but if we know the modules are well behaved, we no longer have to throw everything but the kitchen sink at it.

With small, simple modules, static analysis tools like Sonarqube can help you analyze your test coverage, and make sure modules are not too complicated to adequately test. You also might look at the techniques that teams who write safety-critical software use, like The Power of 10: Rules for Developing Safety-Critical CodeSome of their rules, like no dynamic memory allocation, are too stringent for a lot of products, but it is worth looking at what they do.

Another area where design can improve testing is separating data from its presentation, as in the Model-View-Controller, Model-View-Presenter, or Model-View-ViewModel design patterns. Rather than having to test via screen-scraping tools that work with the GUI, you can test at the Model API, which makes it a lot easier to automate tests. You still have to test the View and Controller, Presenter, or ViewModel to make sure they accurately reflect the Model, but the effort is a lot less.

Debugging flags can be helpful too. I mentioned them here in the context of shipping with partially-implemented features, but they are also useful for displaying information that the customer normally does not want to see, but can be useful for debugging and testing. When the customer calls in with a problem, you tell them to turn on this debugging flag and send you the output, which makes it a lot easier to triage the problem remotely. And during testing, you can turn them on to see intermediate results that help you know the product is operating properly. And since you have followed the Single-Responsibility Principle, you can be pretty confident that turning on the debugging flags will not suddenly make the product act different.

Finally, automated test runners, as have been made popular by Test-Driven Development, let you run tests automatically whenever a module changes. Some examples of these are Jenkins (formerly known as Hudson) and TeamCity.

The Quality Control Inspector of the 1970s has given way to the Quality Assurance Designer of the new millennium, who is less concerned with finding bugs, and more concerned with making sure they do not happen.

This entry was posted in Uncategorized. Bookmark the permalink.

Comments are closed.