Post a Comment
News and New Products
Preventing bug escapes: Panel ponders verification
By Michael Santarini, Senior Editor -- EDN, 2/8/2006
Having talented design and verification engineers on your staff is the No. 1 way to reduce the chances a fatal bug will escape detection and require a respin or make its way to production silicon. But having a comprehensive verification methodology, not just lots of tools, is also important, according to design managers and executives from EDA and IP companies who spoke Tuesday on a DesignCon panel entitled "Managing Verification ROI: Assessing the Business Impact of Bug Escapes."
Moderated by former Intel and Cadence executive Lucio Lanza, now managing director of Lanza techVentures, the panel included MIPS CEO John Bourgoin, Tundra Semiconductor vice president of engineering Benny Chang, Mentor Graphics verification and test group general manager Robert Hum, Jasper Design Automation CEO Kathryn Kranen, IBM distinguished engineer Wolfgang Roesner, and Cisco Systems director of engineering for advanced routing technology Nader Vasseghi.
Panelists said that verification consumes anywhere from 55% to upwards of 80% of the overall chip-design effort, yet there is no way today to ensure 100% of a design has been verified, nor is there even a way to tell that you have performed enough verification. And all that keeps design managers up at night.
"We realize that we can't be perfect," Chang said. "The chips today are large and complex and have a lot of complex interactions on chip. They get used in many many different ways by many different customers and in ways that we as the chip provider never imagined when we first produced the chip."
Bourgoin echoed that sentiment, saying the problem is even more acute for IP companies like MIPS, which builds cores that get used in a broad range of applications. "There are 29 million ways someone can configure our processors," Bourgoin said. "Forget about software, I'm just talking about hardware configuration—there are 29 million ways to do it."
Missing a critical bug can lead to a respin and perhaps put a startup or frail company out of business. In the case of MIPS, the core has to be bug free, work out of the box, and be accessible to a broad user base with a widely varying skill set. "For the IP business, quality is a big differentiator and ultimately determines success," Bourgoin said.
The later in the design process you find a bug, the more expensive it tends to be, Chang and other panelists said.
Bugs usually come in three categories, Vasseghi said:
- Bugs that have a software workaround
- Bugs that have a minor impact on performance or behavior and may ultimately be tolerable or require a layer respin just to be safe, and
- Bugs that are showstoppers, which absolutely require a respin.
"The third one absolutely needs to be fixed," he said. "A silicon respin is required and respinning may have an impact on release schedule, as well as dollars spent and resources."
Through the lifetime of the verification project, IBM monitors two main parameters that deal with the productivity of finding bugs, Roesner said. "One is the steepness of the bug-finding curve and the other one is really the amount of hard-to-detect bugs that really escape to the labs," Roesner said. "We are not living under the illusion with our chips that we will find the last bug in the end, so we have to have a strategy to deal with escapes."
The panelists, even the EDA vendors, concurred that all verification techniques are useful in their own ways. And they encouraged users to employ a mix of emulation/acceleration/emulation, assertion-based verification, and formal-verification methods.
In observing verification projects, IBM has found that lab and systems test are the most effective raw engines for detecting bugs, followed by formal verification, design, acceleration and emulation, chip simulation, and system simulation, Roesner said.
"Early in the design is where we catch the bugs easiest," he said. "We have less efficiency in unit and chip verification and in system verification where we put the system together, where we have the least effective raw engine, but we still find bugs. Formal verification, the next big promise in this area, is very effective in reaching deep into this state space if you can employ it. Acceleration and emulation is very important because you have a fast engine with its own constraints. Finally, when you get silicon back from the lab is where you have the raw power of physics going on and can finish the debug cycle."
Finding bugs in model testing is the least expensive and most desired approach, Bourgoin said. But the cost of a bug goes up 10× if it's detected in component test, 10× more if it's discovered in system test, and 10× more if it's discovered in the field, leading to a failure, a recall, or damage a customer's reputation, he added.
That's why MIPS spares no expense trying to catch bugs early in the model-testing phase, he said. "The more you can do here the more effective you are, and that's something we spend a lot of time on. In fact there is almost no tool or person we won't hire if we believe we can get a material improvement in this space."
Perhaps the most important improvement users can make to their verification ROI is to invest in the design process to proactively prevent designers from writing buggy code, the panelists agreed.
"We invest very heavily in improving the code quality so we reduce the number of bugs the verification teams find, as opposed to relying on the verification team to find bugs for us," Chang said.
Vasseghi seconded that, saying that better design planning and exploration tools at the front end help to ensure designers introduce fewer bugs into designs.
Panelists strongly agreed that new methodology development is the key to improving bug catching. They suggested that tool and methodology innovation requires EDA vendors to work closely with their customers and standards bodies.
Methodology is "ultimate" and tools are "proximate," and tools and methodology must evolve together, Hum said. Working closely with customers helps EDA vendors develop new tools and methodologies, while standards efforts help solidify and proliferate them throughout the design community. "Ultimately, tools are just tools," Hum said. "Methodology creates effectiveness; tools create efficiency."
While panelists seemed fairly open to trying anything that would help find bugs, especially early in the process, they recognized that moving to a new methodology poses a conundrum to users: Will the new method be effective, or will it waste time and perhaps even be more error prone than older verification methods?
The two representatives of the EDA industry on the panel said the rewards of trying new methods ultimately outweigh the risk. Indeed, Kranen suggested that if ultimately designers could be certain of their verification results, they could add that newest feature to their design, make late-stage spec changes, implement newer, more innovative architectures, and perhaps leapfrog their competitors.
