HW/SW co-verification basics: Part 4 - Co-verification metrics - Embedded.com

HW/SW co-verification basics: Part 4 – Co-verification metrics

Many metrics can be used to determine which of the co-verification methods discussed in Part 1, Part 2 and Part 3 are best for a particular project including: 1) Performance (speed); 2) Accuracy; 3) Synchronization; 4) Type of software to be verified; 5) Ability to do hardware debugging (visibility); 6) Ability to do performance analysis; 7) Specific versus general-purpose solutions; 8) Software only (simulated hardware) versus hardware methods; 9) Time to integrate software debug tools; 10) Pre-silicon compared to post-silicon; and, finally, 11) Time to create & integrate models: bus interface, cache, peripherals, RTOS.

Co-verification performance
It is common to see numbers thrown out about cycles/sec and instructions/sec related to co-verification. While some projects may indeed achieve very high performance using co-verification, it is difficult to predict performance of a co-verification solution.

Of course, every vendor will say that performance is “design dependent,” but with a good understanding of co-verification methods it is possible to get a good feel for what kind of performance can be achieved.

The general unpredictability is a result of two factors; first, many co-verification methods use a dual-process architecture to execute hardware and software. Second, the size of the design, the level of detail of the simulation, and the performance of the hardware verification platform results in very different performance levels.

Verification Accuracy
While performance issues are the number one objection to co-verification from software engineers, accuracy is the number one concern of hardware engineers. Some common questions to think about when evaluating co-verification accuracy are listed here. The key to successful hardware/software co-verification is the microprocessor model.

1) How is the model verified to guarantee it behaves identically to the device silicon?
Software models can be verified by using manufacturing test vectors from the microprocessor vendor or running a side-by-side comparison with the microprocessor RTL design database. Metrics such as code coverage can also provide information about software model testing.

Alternatively, not all co-verification techniques rely on separately developed models. Techniques based on RTL code for the CPU can eliminate this question altogether. Make sure the model comes with a documented verification plan. Anybody can make a model, but the effort required to make a good model should not be underestimated.

2) Does the model contain complete functionality including all peripherals?
Using bus functional models was a feasible modeling method before so many peripherals were integrated with the microprocessor. For chips with high integration, It becomes very difficult to model all of the peripherals. Even if a device appears to have no integrated peripherals, look for things like cache controllers and write buffers.

3) Is the model cycle accurate?
Do all parts of the model take into account the internal clock of the microprocessor`? This includes things such as the microprocessor pipeline timing and the correlation of bus transaction times with instruction execution. This may or may not be necessary depending on the goals of co-verification. A noncycle accurate model can run at a higher speed and more may be more suitable for software development.

4) Are all features of the bus protocol modeled?
Many microprocessors use more complex bus protocols to improve performance. Techniques such as bus pipelining, bursting, out-of-order transaction completion, write posting and write reordering are usually a source of design errors. Simple read software. Second, the size of the design, the level of detail of the simulation, and the performance of the hardware verification platform results in very different performance levels.

Verification Accuracy
While performance issues are the number one objection to co-verification from software engineers, accuracy is the number one concern of hardware engineers. Some common questions to think about when evaluating co-verification accuracy are listed here. The key to successful hardware/software co-verification is the microprocessor model.

1) How is the model verified to guarantee it behaves identically to the device silicon?
Software models can be verified by using manufacturing test vectors from the microprocessor vendor or running a side-by-side comparison with the microprocessor RTL design database. Metrics such as code coverage can also provide information about software model testing. Alternatively, not all co-verification techniques rely on separately developed models.

Techniques based on RTL code for the CPU can eliminate this question altogether. Make sure the model comes with a documented verification plan. Anybody can make a model, but the effort required to make a good model should not be underestimated.

2) Does the model contain complete functionality, including all peripherals?
Using bus functional models was a feasible modeling method before so many peripherals were integrated with the microprocessor. For chips with high integration. it becomes very difficult to model all of the peripherals. Even if a device appears to have no integrated peripherals, look for things like cache controllers and write buffers.

3) Is the model cycle accurate?
Do all parts of the model take into account the internal clock of the microprocessor? This includes things such as the microprocessor pipeline timing and the correlation of bus transaction times with instruction execution. This may or may not be necessary depending on the goals of co-verification. A noncycle accurate model can run at a higher speed and more may be more suitable for software development.

4) Are all featue of the bus protocol modeled?
Many microprocessors use more complex bus protocols to improve performance. Techniques such as bus pipelining, bursting, out-of-order transaction completion, write posting, and write reordering are usually a source of design errors.

Simple read and write transactions by themselves rarely bring out hardware design errors. It is the sequence of many transactions of different types that bring out most design errors. There is nothing more frustrating than trying to use a model for a CPU that has multiple modes of operation only to find out that the mode that is used by the design suffers from the most dreaded word in modeling, “unsupported.”

I once worked on a project involving a design with the ARM920T CPU. This core uses separate clocks for the bus clock (BCLK) and the internal core clock (FCLK). The clocking has three modes of operation:

FastBus Mode : The internal CPU is clocked directly from the bus clock (BCLK) and FCLK is not used.
Synchronous Mode : The internal CPU is clocked from FCLK, which must be faster and a synchronous integer multiple of the bus clock (BCLK).
Asynchronous Mode: The internal CPU is clocked from FCLK, which can be totally asynchronous to BCLK as long as it is faster than BCLK.

As you can probably guess the asynchronous mode caused this particular problem. The ARM920T starts off using FastBUS mode after reset until which time software can change a bit in coprocessor 15 to switch to one of the other clocking modes to get higher performance.

When the appropriate bit in cpl5 was changed to enable asynchronous mode, a mysterious message comes out: “Set to Asynch mode, WARNING this is not supported. ” It is quite disheartening to learn this information only after a long campaign to convince the project team that co-verification is useful.

Following are some other things to pay attention to when evaluating models used for co-verification:

1) Can performance data he gathered to ensure system design meets requirements? If the model is not cycle accurate, the answer is NO. Both hardware and software engineers are interested in using co-verification to obtain measurements about bus throughput, cache hit rates, and software performance. A model that is not cycle accurate cannot provide this information.

2) Will the model accurately model hardware and software tinting issues? Like the bus protocol, the more difficult to find software errors are brought out by the timing interactions between software and hardware. Examples include interrupt latency. timers, and polling. When it comes to modeling and accuracy issues there are really two different groups.

One set of engineers is mostly interested in the value of co-verification for software development purposes. If co-verification provides a way to gain early access to the hardware and the code runs at a high enough speed the software engineer is quite happy and the details of accuracy are not that important.

The other group is the hardware and verification engineers who insist that if the simulation is not exact then there is no reason to even bother to run it. Simulating something that is not reality provides no benefit to these engineers. The following examples demonstrate the difficulty in satisfying both groups.

AHB Arbitration and Cycle Accuracy Issues
I once worked with a project that used a single-layer AHB implementation for the ARM926EJ-S. One way to perform co-verification is to replace a full-functional logic simulation model of the ARM CPU by a bus functional model and an instruction set simulator.

Since the ARM926 bus functional model will actually implement two bus interfaces, a question was raised by the project team about arbitration and the ordering of the transfers on the bus between the lAHB and the DAHB. The order in which the arbiter will grant the bus is greatly dependent on the timing of the bus request signals from each AHB master.

From the discussion of AHB the HREADY signal plays a key role in arbitration since the bus is only granted when both HGRANT and HREADY are high. The particular bus functional model being used decided that since HREADY is required to be high for arbitration and data transfer, it can be used more like an enable for the bus interface model state machine since nothing can happen without it being high.

To optimize performance the bus functional model decided to do nothing during the time when HREADY was low. This assumption about the function of HREADY is nearly correct, but not exactly.

The CPU indication to the bus interface unit of the ARM926 that it needs to request the bus has nothing to do with HREADY or the bus interface clock, it uses the CPU clock. This produced a situation where the IHBUSREQ and DHBUSREQ were artificially linked to the HREADY and the timing of these was incorrect.

The result to the user was the arbiter granting the IAHB to use the bus instead of the DAHB. Since the two busses are independent, except for the few exceptions we discussed, there is no harm in running the transactions in a different order on the single-layer AHB.

Functionally, this makes no difference and all verification tests and software will execute just fine, but to hardware engineers seeking accuracy the situation is no good. This case does bring up some interesting questions related to performance: 1) Does arbitration priority affect system performance? and 2) What are the performance differences between single-layer AHB versus multilayer AHB versus separate AHB interfaces?

Figure 6.24 below shows the correct bus request timing. The sequence shows the IAHB reading addresses 0x44 and 0x48 followed by the DAHB reading from address 0x90. It is difficult to see. but the next transaction is the IAHB reading from address Ox4c. 

Figure 6.24: Correct Timing of Bus Request (To view larger image, click here)

Notice the timing of DHBUSREQ at the start of the waveform. It transitions high before the first IHREADY on the waveform. This demonstrates that the timing of DHBUSREQ is not related to HREADY.

Figure 6.25 below shows the incorrect ordering of bus transfers caused by the difference in the timing of DHBUSREQ. The sequence start the same way with IAHB reads from 0x44 and 0x48, but the read from Ox4c comes before the DAHB read from address 0x90. 

Figure 6.25: Incorrect Timing of Bus Request (To view larger image, click here)

The reason is the timing of DHBUSREQ. Notice DHBUSREQ transitions high AFTER the first IHREADY on the waveform. This difference results in out-of-order transactions. Contrast this pursuit of accuracy with a software engineer I met once that didn't care anything about the detail of the hardware simulation. Skipping the majority of the simulation activity just to run fast was the best way to go.

He had no desire to run a detailed, cycle-accurate simulation. Actually, he was interested in making sure the software ran on the cycle-accurate simulation, but once it had been debugged using a noncycle accurate co-verification environment the final check of the software was better suited for a long batch simulation using the ARM RTL model and farm of workstations that was maintained by the hardware engineers, not him.

Since the chance of finding a software bug was low there was no reason to worry about the problem of debugging the software in a pure logic simulation environment using waveforns or logfiles.

Modeling Summary
Modeling is always painful. There is no way around it. No matter what kind of checks and balances are available to compare the model to the actual implementation, there are always differences. One of the common debates is about what represents the golden view of the IP. In the case of ARM microprocessors, there are three possible representations that are considered “golden”:

1) RTL code (for synthesizable ARM designs)
2) Design sign-off model (DSM) derived from the implementation
3) Silicon in the form of a test chip or FPGA

Engineers view these three as golden, even more so than the specification. Co-verification techniques that use models that do not come from one of these golden sources are at a disadvantage since any problems are always blamed on the “nongolden” model.

I have seen cases where the user's design does not match the bus specification, but does work when simulated with a golden model. Since a specification is not executable, engineers feel strongly that the design working with the golden model is most important, not the specification.

When alternative models are used for co-verification a model that conforms to the specification is still viewed as a buggy model in any places it differs from the golden model. It is not always easy to convince engineers that a design that runs with many different models and adheres to the specification is better than a design that runs only with the golden model.

Synchronization
Most co-verification tools operate by hiding cycles from the slower logic simulation environment. Because of this, issues related to synchronization of the microprocessor with the rest of the simulated design often arise.

This situation is also true using in-circuit emulation with a processor linked to the emulator via a speed bridge. In co-verification there are two distinct time domains, the microprocessor model running outside of the logic simulator and the logic simulator itself.

Understanding the correlation of these two time domains is important to achieving success with co-verification. Co-verification uses mainly the spatial memory references to decide when software meets the hardware simulation.

Synchronization is defined by what happens to the logic simulator when there is no bus transaction occurring in the logic simulator. It could be stopped until a new bus transaction is received. It could just “drift” forward in time executing other parts of the logic simulation hardware (even with an idle microprocessor bus ).

In either case the amount of time simulated in the logic simulator and the microprocessor model is different. Another alternative is to advance the logic simulation time the proper number of clock cycles to account for the hidden bus transaction but don't run the transaction on the bus.

Now the correction of the time domains is maintained at the expense of performance. Synchronization is also important for those temporal activities where the hardware design communicates with software such as interrupts and DMA transfers. Without proper synchronization, things like system timers and DMA transfers may not work correctly because of differences in the two time domains.

Types of Software
The type of software to be verified also has a major impact on which co-verification methods to deploy. There are different types of software that engineers see as candidates for co-verification: system diagnostics, device drivers. RTOS, and application code.

Different co-verification methods are better suited to different types of software. Usually the lower level software requires a more accurate co-verification environment and higher-level software is less interested in accuracy and more focused on performance because of the code size.

Running an RTOS such as VxWorks has been shown to be viable by multiple co-verification methods including in-circuit emulation, an ISS, and the RTOS simulator, VxSfM. Even with the marketing claims that software does not have to be modified, expect some modification to optimize such things like long memory tests and DART accesses.

The major confusion today exists because of the many types of software and the many methods of hardware execution. Often, different levels of performance will enable different levels of software to be verified using co-verification.

A quick sanity check to calculate the number of cycles required to run a given type of software and the speed of the environment will ensure engineers can remain productive. If it takes 1 hour to run a software program to get to the new software this means the software engineer will have only a handful of chances per day to run the code and debug any problems found.

Still more co-verification metrics
Besides performance and accuracy, there are some other metrics worth thinking about. Project teams should also determine if a general-purpose solution is important versus a project specific solution.

General-purpose solutions can be reused on future projects and only one set of tools needs to be learned. Unfortunately, general-purpose solutions are not general if the model used on the next project is not available.

Methods using the evaluation board or prototyping are more specific and may not be applicable on the next project. For many engineers, especially software engineers, a solution that consists of simulation only is preferred over one that contains hardware.

Another important distinction is whether the solution is available pre- or post-silicon. Many leading edge projects use microprocessors that are not yet available and a pre-silicon method is required. All of these variables should be considered when deciding on a co-verification strategy.

Understanding all of these metrics will avoid committing to a co-verification solution that will not meet the project needs. Remember, the goal of co-verification is to save time in the project schedule.

To read Part 1 , go to “ Determining what and how to verify .
To read Part 2, go to “ Software-centric co-verification methods.”
To read Part 3 , go to “Hardware-centric co-verification methods.”

This series of articles by Jason Andrews is from “Embedded Software know it all” edited by Jack Ganssle, used with permission from Newnes, a division of Elsevier. Copyright 2008. For more information about this title and other similar books, please visit www.elsevierdirect.com.

Jason Andrews, author of Co-verification of Hardware and Software ARM SoC Design , has implemented multiple commercial co-verification tools as well as many custom co-verification solutions. His experience in the EDA and embedded marketplace includes software development and product management at Verisity, Axis Systems, Simpod, Summit Design. and Simulation Technologies. He has presented technical papers and tutorials at the Embedded Systems Conference, Communication Design Conference, and IP/SoC and written numerous articles related to HW/SW co-verification and design verification. He has a B.S. in electrical engineering from The Citadel, Charleston, S.C., and an M.S. in electrical engineering from the University of Minnesota.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.