My First Example with OS-VVM CoveragePkg

Here in Europe, I recently had the opportunity to work with Jim Lewis, OS-VVM Chief Architect and IEEE 1076 Working Group Chair, on the first Advanced VHDL Testbenches & Verification training course. This training, held in Bracknell, UK, was attended by engineers from several major European system companies who design and verify programmable devices (FPGAs). VHDL is by far the dominate language used by Europe’s system companies for the design and verification of FPGAs, however it is unclear to many how to enhance their verification with VHDL. What I have found is that experienced FPGA design engineers (including myself) are not utilising the VHDL language for verification.

Verification with VHDL

Jim Lewis introduces VHDL’s verification capabilities, including new VHDL 2008 features and the Open Source VHDL Verification Methodology (OSVVM). OSVVM provides a methodology for testbench development and verification packages that provide functional coverage and random value generation.

To demonstrate the use of the OSVVM functional coverage package ‘CoveragePkg’, I took a familiar Finite-state Machine (FSM) the IEEE 1149.1 TAP (Test Access Port) controller. This is a clever little state machine that I am rather fond of, having spent a lot of time working with it as part of my degree dissertation on Built-in Self Test (BIST).

So let’s take a look at our IEEE 1149.1 TAP FSM.

The TAP FSM transitions are controlled by the Test Mode Select (TMS) signal.  The TAP controls the JTAG system performing serial shift and parallel load of the data registers (DR) and instruction registers (IR).

Figure 1: TAP State Machine

Code Coverage vs Functional Coverage

To explain what functional coverage is and the value in terms of verification, let’s first take a look at code coverage.

Code coverage is a technique that uses instrumentation to record the execution of the code within the simulator.  Two well-known techniques are statement and branch coverage.  Statement coverage tells you which statements have been executed in simulation i.e. “hit”.  Branch coverage provides information about the branches taken of control structures such as ‘if-then-else’ statements.

So let’s run my example test scenario for the TAP FSM and take a look at the code coverage reports. 

Figure 2: Sample Code Coverage Reports

In Figure 2, statements that have not been “hit” are highlighted in red. From the report we can see that our test scenario does not test (cover) all the state transitions. This is what code coverage provides; it tells you what has not been tested, and is a very valuable metric for any verification process.  

However, we would be wrong to say we have tested (covered) all the “green” state transitions.  Why?  Code coverage records the execution of the code not the function and so is sensitive to delta cycles.  Let’s work through an example to illustrate what can happen.

Looking at the statement coverage report, you might think that we have exercised the two transitions of the SHIFT_DR state (i.e. SHIFT_DR to SHIFT_DR, and SHIFT_DR to EXIT1_DR).   For the test scenario the FSM is only ever in SHIFT_DR for one clock cycle.  So what happened?  Let’s take a look at the waveform (Figure 3) with the delta cycles enabled.

Figure 3: Waveform showing delta cycles

The TAP FSM consists of two VHDL processes; combinatorial next state logic and a sequential state register.   For this simple example I added a concurrent signal assignment (TMS_SIG) to TMS so as to introduce a delta cycle.  What has happened is the combinatorial next state process was run twice, once with TMS_SIG equal to ‘0’ and a second time with TMS_SIG equal to ‘1’.  This was due to the introduced delta cycle and the sensitivity list of the next state process.

The execution in the simulator is shown in Table 1.

Table 1: The execution of signal evaluation and update in the simulator

So the TAP FSM next state logic settled on EXIT1_DR.  The functional state transition that took place was SHIFT_DR to EXIT1_DR.  However, in the simulation delta cycles 2 and 3 we executed each branch of the if-then-else statement for the SHIFT_DR next state logic.

This is the difference between functional coverage and code coverage. Functional coverage collects the actual state transitions (the function) so as to build up a complete picture of how the FSM has been functionally exercised. This is achieved with a coverage model.  

Adding Functional Coverage with OS-VVM

Creating a coverage model with the OS-VVM package ‘CoveragePkg’ turned out to be very simple.  Let’s walk through the steps to create a coverage model for the TAP FSM. First you need the OS-VVM packages, these are included in the Aldec simulator and are available for download on the OS-VVM website.

Next we need to create a coverage object for the FSM, which is a protected type from ‘CoveragePkg’ and used to collect and report the functional coverage.

So to observe the TAP FSM state register in the testbench we use a VHDL 2008 external name and alias this to the TAP_STATE.  

The TAP FSM uses an enumerated type which is defined in the package ‘TapTypes’.

Next we use a process to define the valid state transitions and collect data - this is the coverage model for our TAP FSM.   

We can give our coverage object a unique name which is then used for reporting.

To define the state transitions we use cross coverage which defines relationships between multiple objects, in this case the current and previous state of our TAP FSM. To do this we need to capture the state transitions in tabular form with ‘current’ and ‘next state’.  You could extract the FSM from the RTL as this is a relatively straightforward task, assuming you have a defined FSM coding style.  However as a verification model it would be considered good practice to use the original specification.

VHDL is a very rich language and for the enumerated type ‘TapStateType’ we can use the predefined type attributes.  The coverage model is based on integer bins so we use the T’POS attribute to define an integer value for each state.

Once we have defined all the valid state transitions we mark the rest of the possible transitions as illegal.

Next we need to define our collector.  This will sample on the negative edge of the TCK clock and will keep track of the previous state.  The call of the protected type procedure ICover (method) accumulates the coverage.   We call ICover with an integer vector that represents the state transition.

The last step is to generate a coverage report at the end of simulation.

The reports shows a 78.125% coverage for the test scenario and as expected the SHIFT_DR -> SHIFT_DR transition is not covered. The report also confirms that the state PAUSE_DR was not covered.

The test scenario that I am running is using another OS-VVM package ‘RandomPkg’ to stimulate the TAP FSM with a pseudo-random sequence for TMS.   The test scenario was run with a target of >75% coverage to demonstrate the coverage holes.  If we change this to 100% and rerun we will now hit 100% coverage in 119 cycles.

Summary

We looked at the difference between code coverage and functional coverage and how to add functional coverage with VHDL OS-VVM ‘CoveragePkg’ to a FSM. Often customers raise concerns about the amount of work required to create a complete coverage model for a design. This can be significant and is one of the reasons why ASIC teams often have dedicated verification engineers.

With programmable devices, a complete coverage model is generally not as critical (unless the application is safety critical).  Using OS-VVM ‘CoveragePkg’ you can incrementally introduce functional coverage to your test environment to analyse your current test scenarios in terms of functional coverage.  This has the immediate benefit of providing additional coverage metrics for your current verification.  These metrics can be used to identify areas for improvement.   For example, they could be used to “grade” tests so that test scenarios with the greatest coverage are run first in a regression.

For more on OSVVM, visit www.osvvm.org.