How to Avoid “Firefighting” in Verification

Misstep #1: Insufficient planning

Insufficient planning occurs when you don’t have what you need to do testing, and your test coverage falls short. It’s caused by undocumented assumptions, the increasing scope of projects, and “missed dependencies” (you need 10 prototypes but only get 5). “If you don’t plan for it, it will surprise you, and every surprise will end up as a fire.”

The solution? Put your plan in writing – including who does what, how features work, what it means to be “done,” what checkpoints will monitor progress, and criteria for success. Keeping track of assumptions may be the biggest part of the solution. Write them down!

Misstep #2: Not designing for test

Designers often think their designs won’t have any mistakes, so there’s no plan for testing and no communication with validators. This makes it difficult to find and replicate bugs, to figure out what you need to monitor, and to know when you’re done. Interpreting test results as “pass” or “failure” may be very difficult. The antidote is for validators to get involved in the earliest stages of the design process. “Ask how you’re going to test it and how you’re going to tell if it’s working.”

Misstep #3: Not creating and integrating feedback loops

All too often, the marketing team or the design engineers make changes to a product, and don’t communicate those changes to the verification team. Further, many companies place engineers in “silos” with little or no communication – for example, there are software engineers, hardware engineers, and firmware engineers who don’t talk to each other.

What’s needed is continuous feedback about any changes in the product, as well as problems found with the product. Tests should be monitored for effectiveness and continually improved.

Misstep #4: Lack of transparency

Lack of transparency happens when you tell your boss (or team) that everything is well when it really isn’t. Or, you skimp on tests and coverage as schedule pressure rises, and don’t let managers know. As a result, risks and coverage gaps increase. “Tell the real story, and encourage others to do the same. Don’t declare that it’s done until it’s really done.”