mciepluc / cocotb-coverage Goto Github PK
View Code? Open in Web Editor NEWFunctional Coverage and Constrained Randomization Extensions for Cocotb
License: BSD 2-Clause "Simplified" License
cocotb-coverage's People
Contributors
Stargazers
Watchers
Forkers
cmarqu pwiecha themperek yqsang sjalloq loesterfranco benbr8 psumesh incoresemi imphil jonpovey glkclass gyx3598 danielaiyuancocotb-coverage's Issues
Add YAML coverage database import/export
Export coverage database to yaml format, and import external yaml to the CoverageDB object.
Related #22
Add Sphinx code to handle epydoc formatting in constraint.py
adder_test_coverage
How to display coverage number and trend from XML data
What would be the recommended tool/plug-in to be able to display current functional coverage numbers and coverage trend from the generated XML file in Jenkins? Does the XML data need to be parsed by another tool before it can be picked up by Jenkins plugins like JUnit or Cobertura? Please make me smart as I am not familiar with the post sim flow. Thank you in advance.
How to write distributions
Hi,
I was just trying to write a test case for a distribution that doesn't seem to work and found a bunch of examples in your tests that I have no clue about.
What does the following constraint do in your test_distributions_1?
d2 = lambda y: 2 * y
Is there some implied missing syntax here because it doesn't seem to make sense.
Then, for a standard distribution, I was trying to force a 50/50 distribution of reads/writes in my transaction but it seems to tend towards 33:66. Is the following not correct?
dist = lambda rnw: 0.5 if rnw else 0.5 where rnw is my readNotWrite flag.
Support for UCIS coverage format
https://www.amiq.com/consulting/2018/11/15/new-release-of-the-functional-coverage-for-systemc-library/ can write UCIS, merge coverage databases, and has a pretty nice HTML GUI for cumulative coverage results.
https://github.com/amiq-consulting/fc4sc/blob/master/doc/FC4SC_User_Guide.pdf shows the rendered result.
License is Apache v2.
Originally posted by @cmarqu in #22 (comment)
CI setup
Seems travis is not working anymore... need to setup a contiuous integration with already existing unit tests.
CoverPoint and CoverCross classes don't support inheritance
Following code leads to problems with class iheritance:
def __new__(cls): return super(CoverPoint, cls).__new__(CoverPoint)
Better to use:
def __new__(cls): return super(CoverPoint, cls).__new__(cls)
which should make code more generic.
Various improvements suggestions
I do have a little bit feedback.
Coverage
- Explicitly named bins would ease reading of the coverage report.
coverage.CoverPoint("transaction.payload_length",
...
bins = [('runt', (0, 64)), ('normal', (65, 1500)), ('giant', (1501, 10000))]),
...
- Promote coverage_db to a true singleton object with various query and reporting methods. Arguments include filtering by bin or transaction name, amount of detail, etc. Also a custom accessor to get and inspect a particular CoverItem. This could allow feedback to CRV to achieve faster functional coverage during runtime.
coverage_db = cocotb.coverage.get_coverage_database()
coverage_db.report_covered(**kwargs)
coverage_db.report_uncovered(**kwargs)
coverage_db.get_coverage_for_item(item_name)
# etc.
CRV
- A standardized method to build random lists, such as for packet payloads. Would take parameters for min/max
len()and min/max values that applies to all values in the list. I realize the underlying constraint.py doesn't support this. Maybe just implement using random module and generate duringpre_randomize()orpost_randomize()? Other constraints could then use len(txn.payload) in the constraint solver.
I hacked together something like this in my base transaction class:
from random import choice
from cocotb.crv import Randomized
class MyRandomized(Randomized):
def __init__(self):
super().__init__()
self._rand_lists = {}
def add_rand_list(self, name, lengths, values, pre_randomize=True):
self._rand_lists[name] = (lengths, values, pre_randomize)
setattr(self, name, [])
def build_rand_lists(self, pre_randomize):
for name, (lengths, values, pr) in self._rand_lists.items():
if pre_randomize == pr:
setattr(self, name, [choice(values) for x in range(choice(lengths))])
def pre_randomize(self):
self.build_rand_lists(pre_randomize=True)
def post_randomize(self):
self.build_rand_lists(pre_randomize=False)
- A small bit of style advice. Use CamelCase for class names but all_lower_case() for methods. E.g.,
addRand()-->add_rand()
Keep up the good work!
Originally posted by @jrpetrus in cocotb/cocotb#490 (comment)
Documentation improvements
As this project is mainly addressed for SystemVerilog and commercial tools users to encourage them to abandon the sinking ship, it is good to provide a more informative examples and tutorials. The goal is to show consistency with SystemVerilog coverage/randomization features with much higher flexibility and user-friendliness.
Also @cmarqu provided an interesting tutorial: https://www.divio.com/blog/documentation/
Please share any suggestions what should be included in the documentation.
Some suggestions:
- FIFO example documentation
- add more examples, focus on CRV (simple protocol, simple packet router?)
- tutorial how to translate SV syntax to cocotb-coverage
- tutorial what you can't do (easily) in SV and can do in cocotb-coverage
Ignore bins: dictionary changed size during iteration
Adding ignore bins to CoverCross instance gives a runtime error mentioned in the topic. The culprit is coverage.py, line 342, for x_bins in self._hits.keys():. According to SO https://stackoverflow.com/a/11941855/2922946 in Python 3 keys returns an iterator instead of a list.
Encapsulating with list solved this problem for me list(self._hits.keys()).
Seed setup
How do I set up a seed value and make sure the the randomization consistency is stable for every run and will get the same randomized results with the same seed?
Thanks,
Richard
Packet swtich example
Add another example.
cocotb-coverage release and package
At some point when documentation and cleanup work is done, it is nice to release the package and make it pip installable (the way like cocotb core). Help wanted @themperek :)
randomize() function generates repeated values
How to generate non repetitive values using randomize() function in crv???
Move from unittest to pytest?
With cocotb-test, which will probably move into cocotb in some form, the use of unittest looks a bit unusual.
What do you think about converting the tests to pytest? (There is https://github.com/pytest-dev/unittest2pytest which may be of use.)
Failed to import module test_fifo: 'return' with argument inside generator (test_fifo.py line 133)
Hi,
I downloaded the lib and ran make SIM=icarus -d because it failed import test_fifo.py due to Syntax error. I didn't do any modification. Here is the long version of the message. Please help. Thanks.
0.00ns INFO cocotb.regression regression.py:133 in initialise MODULE variable was "test_fifo"
0.00ns INFO cocotb.regression regression.py:134 in initialise Traceback:
0.00ns INFO cocotb.regression regression.py:135 in initialise Traceback (most recent call last):
File "/home/dywcheng/nonShareVM/cocotb/cocotb/regression.py", line 130, in initialise
module = _my_import(module_name)
File "/home/dywcheng/nonShareVM/cocotb/cocotb/regression.py", line 65, in _my_import
mod = __import__(name)
SyntaxError: 'return' with argument inside generator (test_fifo.py, line 133)
24k downloads per month 😮
Hello cocotb-coverage users,
I can see a crazy number of 24k downloads of this package per month. Compared to core cocotb (which has 60k), it is a 40%. I am not sure if this number is valid, and probably increased by many CI tasks running daily, but I am still impressed. It was around 1k one year ago. I would be verry happy if you could write a few words about how you use this package and if it is useful. Also please feel free to contribute and propose new features. Thank you all!
Cocotb coverage simple adder test
Exception using randomize_with
I'm seeing an exception raised when I try to randomize an object. I have a class that models an AHB transaction and I'm trying to randomize the number of cycles I stall before driving it on the bus. So my transaction looks something like:
class Transaction(Randomized):
def __init__(self, address, etc):
self.address = address
etc
self.delay = 0
self._delay = "small"
self.add_rand("_delay", ["none", "small", "large"])
def post_randomize(self):
if self._delay == "none":
self.delay = 0
elif self._delay == "small":
self.delay = 5
etcThen when I try to generate a transaction using:
trxns = [Transaction(addr) for addr in range(10)]
for t in trxns:
t.randomize_with(lambda _delay: _delay == "none")I get:
Traceback (most recent call last):
File "test_random.py", line 63, in _run
self._generate_transactions()
File "test_random.py", line 77, in _generate_transactions
self._trxns = self._region.generate_transactions(
File "testbench.py", line 154, in generate_transactions
t.randomize_with(lambda _delay: _delay == "none")
File ".venv/lib/python3.9/site-packages/cocotb_coverage/crv.py", line 339, in randomize_with
raise Exception("Could not resolve implicit constraints!")
Exception: Could not resolve implicit constraints!
What am I doing wrong?
Is this library still being developed or are there better alternatives now? I always seem to hit these issues when I come back to use it.
Thread-safe Singleton
Is coverage database implemented as thread-safe singleton? This may be important when dealing with parallel simulations.
Documentation typos
Examples:
(which depends on the FIFO satus)
At each successfull read from the FIFO, the data concistency
uppoer size limit for length based filtering
If the packed is not filtered or opion x2
packed few times
Tutorials:
There is no a straightforward obsolete "a"
Save coverage database in XML format
coverage_db object should be exportable to XML format.
There should be also an option for merging coverage databases (for sure something like this exists).
Examine also other coverage databases formats, if it makes sense (UCIS?).
Suggest add_rand() creates attribute if it does not exist
I have been reading the documentation (and source) for Randomized and add_rand()
It seems like add_rand() could create an attribute if it is not found in the object, and perhaps initialise it to the first element in the domain list.
It seems like this would reduce boilerplate in common code where currently the examples require writing self.x = some_dummy_value before add_rand("x", ...)
coverage_section build works only with Verilator and not with Icarus
Cocotb simulation builds correctly with Verilator and generate a good coverage report, but it fails to build with Icarus.
The build with Icarus succeeds if the coverage_section decorator is removed from the @cocotb.coroutine.
I added comments in the code to help debugging (full_adder_tb.py and Makefile).
The provided example is a simple full adder and I am not expecting any RTL or python modelling issues.
Resetting the coverage database for each @cocotb.test
Hi
I'm using cocotb with several test cases per files - something like this:
@cocotb.test()
def test1(dut):
.....
@cocotb.test()
def test2(dut):
.....At the end of each test I'm calling coverage_db.export_to_xml to a different file (one xml file per test)
I noticed that the coverage results are "cumulative" when running all the tests from one make command. (coverage results for test2 include the coverage of test1 )
That makes sense as the coverage_db is a global singleton
Is there a clean way to reset the coverage database at the beginning of each test ?
Sphinx setup for documentation
It would be nice if there was a Sphinx setup for documentation of this extension (even if it is not automatically hooked up to readthedocs).
The automodules Sphinx plugin could be used to easily extract all classes and methods etc., but the following issue from cocotb also applies here then: cocotb/cocotb#715
report_coverage() does not accept a logger object as documented
report_coverage() requires a logger argument documented as:
logger (func) – a logger object.
But it does not accept a "logger object" as described in https://docs.python.org/3/library/logging.html
Actually what it wants is a function such as logger.info or print.
Suggest changing this documentation line to something less confusing, perhaps something like:
logger (func) - a logging function (e.g. logger.info)
Coverage cross from two separate coverage_section's?
Hi,
just starting to use the coverage module and have a question regarding whether it's possible to cross coverpoints that aren't sampled together.
Imagine I have a BusTransaction that I'm sampling and I've got {READ,WRITE} bins. I also have some clock config that I'm sampling where I might have bins for FREQ and CLKDIVRATIO, for example. How would I go about creating crosses for these two coverpoints?
I guess in the example I've given I've got one dynamic transaction and one that is static. I could work around this by creating a specific coverage sampling method that was passed in the bus transaction, then generated local static attributes for the frequency and clock ratio fields. I'd still need a single coverage_section though.
I'm assuming that if you've got two separate transactions that are generated independently, there's no way to sample them together? How is this done in the SV world? It's been a while since I looked at how it's done.
Constraint on fixed-sizes arrays?
Not an issue as much as a feature request and/or guidance to potentially help with such request.
What would be needed to support such a structure:
self.my_very_own_array = [0] * 16
for i in range(16)
self.add_rand(f'my_very_own_array[{i}]', [0, 1])Example FIFO gets unexpected output
Hi, I have installed cocotb, cocotb-coverage, iverilog, and icarus successfully, and also the fifo example can run without error by "make SIM=icarus".
But the result seems to be something wrong.
The output below run by 10 random operations.
I expect that "read" after "write" should read the data in fifo successfully, but it always shows "Data NOT read, fifo EMPTY!".
MODULE=test_fifo TESTCASE= TOPLEVEL=fifo_mem TOPLEVEL_LANG=verilog \
/usr/bin/vvp -M /home/user/.conda/envs/py36/lib/python3.6/site-packages/cocotb-1.4.0.dev0-py3.6-linux-x86_64.egg/cocotb/libs -m libcocotbvpi_icarus sim_build/sim.vvp
-.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:71 in set_program_name_in_venv Did not detect Python virtual environment. Using system-wide Python interpreter
-.--ns INFO cocotb.gpi ../gpi/GpiCommon.cpp:106 in gpi_print_registered_impl VPI registered
-.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:264 in embed_sim_init Python interpreter initialized and cocotb loaded!
0.00ns INFO cocotb __init__.py:146 in _initialise_testbench Running on Icarus Verilog version 0.9.7
0.00ns INFO cocotb __init__.py:153 in _initialise_testbench Running tests with cocotb v1.4.0.dev0 from /home/user/.conda/envs/py36/lib/python3.6/site-packages/cocotb-1.4.0.dev0-py3.6-linux-x86_64.egg/cocotb
0.00ns INFO cocotb __init__.py:170 in _initialise_testbench Seeding Python random module with 1590030689
0.00ns INFO cocotb.regression regression.py:127 in __init__ Found test test_fifo.fifo_test
0.00ns INFO cocotb.regression regression.py:453 in _start_test Running test 1/1: fifo_test
0.00ns INFO ..otb.test.fifo_test.0x2aeb40019080 decorators.py:256 in _advance Starting test: "fifo_test"
Description: FIFO Test
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
2.15ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
2.35ns INFO cocotb.test test_fifo.py:153 in fifo_test Data written to fifo: 3D
2.55ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
2.75ns INFO cocotb.test test_fifo.py:153 in fifo_test Data written to fifo: 1
2.95ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.15ns INFO cocotb.test test_fifo.py:153 in fifo_test Data written to fifo: E
3.35ns INFO cocotb.test test_fifo.py:153 in fifo_test Data written to fifo: 66
3.55ns INFO cocotb.test test_fifo.py:153 in fifo_test Data written to fifo: AD
3.75ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.95ns INFO cocotb.test test_fifo.py:153 in fifo_test Data written to fifo: 42
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top : <cocotb_coverage.coverage.CoverItem object at 0x2aeb3fe18da0>, coverage=17, size=32
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_empty : <cocotb_coverage.coverage.CoverPoint object at 0x2aeb3fe18a90>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_full : <cocotb_coverage.coverage.CoverPoint object at 0x2aeb3fe18978>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_overflow : <cocotb_coverage.coverage.CoverPoint object at 0x2aeb3fe18ef0>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_threshold : <cocotb_coverage.coverage.CoverPoint object at 0x2aeb3fe18e48>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_underflow : <cocotb_coverage.coverage.CoverPoint object at 0x2aeb40019cf8>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rw : <cocotb_coverage.coverage.CoverPoint object at 0x2aeb3fe18a58>, coverage=2, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 6
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXempty : <cocotb_coverage.coverage.CoverCross object at 0x2aeb40019048>, coverage=2, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 6
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXfull : <cocotb_coverage.coverage.CoverCross object at 0x2aeb40019128>, coverage=2, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 6
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXoverflow : <cocotb_coverage.coverage.CoverCross object at 0x2aeb40019160>, coverage=2, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 6
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXthreshold : <cocotb_coverage.coverage.CoverCross object at 0x2aeb400190f0>, coverage=2, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 6
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXunderflow : <cocotb_coverage.coverage.CoverCross object at 0x2aeb40019198>, coverage=2, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 6
3.95ns INFO cocotb.regression regression.py:351 in _score_test Test Passed: fifo_test
3.95ns INFO cocotb.regression regression.py:469 in _log_test_summary Passed 1 tests (0 skipped)
3.95ns INFO cocotb.regression regression.py:538 in _log_test_summary *****************************************************************************
** TEST PASS/FAIL SIM TIME(NS) REAL TIME(S) RATIO(NS/S) **
*****************************************************************************
** test_fifo.fifo_test PASS 3.95 0.09 45.28 **
*****************************************************************************
3.95ns INFO cocotb.regression regression.py:555 in _log_sim_summary *************************************************************************************
** ERRORS : 0 **
*************************************************************************************
** SIM TIME : 3.95 NS **
** REAL TIME : 0.12 S **
** SIM / REAL TIME : 32.68 NS/S **
*************************************************************************************
3.95ns INFO cocotb.regression regression.py:249 in tear_down Shutting down...
The output below run by 10 "read" operations.
I expect that fifo_underflow bin should be True, but instead always False.
MODULE=test_fifo TESTCASE= TOPLEVEL=fifo_mem TOPLEVEL_LANG=verilog \
/usr/bin/vvp -M /home/user/.conda/envs/py36/lib/python3.6/site-packages/cocotb-1.4.0.dev0-py3.6-linux-x86_64.egg/cocotb/libs -m libcocotbvpi_icarus sim_build/sim.vvp
-.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:71 in set_program_name_in_venv Did not detect Python virtual environment. Using system-wide Python interpreter
-.--ns INFO cocotb.gpi ../gpi/GpiCommon.cpp:106 in gpi_print_registered_impl VPI registered
-.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:264 in embed_sim_init Python interpreter initialized and cocotb loaded!
0.00ns INFO cocotb __init__.py:146 in _initialise_testbench Running on Icarus Verilog version 0.9.7
0.00ns INFO cocotb __init__.py:153 in _initialise_testbench Running tests with cocotb v1.4.0.dev0 from /home/user/.conda/envs/py36/lib/python3.6/site-packages/cocotb-1.4.0.dev0-py3.6-linux-x86_64.egg/cocotb
0.00ns INFO cocotb __init__.py:170 in _initialise_testbench Seeding Python random module with 1590030410
0.00ns INFO cocotb.regression regression.py:127 in __init__ Found test test_fifo.fifo_test
0.00ns INFO cocotb.regression regression.py:453 in _start_test Running test 1/1: fifo_test
0.00ns INFO ..otb.test.fifo_test.0x2aaedcd6c0b8 decorators.py:256 in _advance Starting test: "fifo_test"
Description: FIFO Test
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
signal_get: property 18 is unknown
2.15ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
2.35ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
2.55ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
2.75ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
2.95ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.15ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.35ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.55ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.75ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.95ns INFO cocotb.test test_fifo.py:148 in fifo_test Data NOT read, fifo EMPTY!
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top : <cocotb_coverage.coverage.CoverItem object at 0x2aaedcb6b8d0>, coverage=11, size=32
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_empty : <cocotb_coverage.coverage.CoverPoint object at 0x2aaedcb6bac8>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_full : <cocotb_coverage.coverage.CoverPoint object at 0x2aaedcb6b9b0>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_overflow : <cocotb_coverage.coverage.CoverPoint object at 0x2aaedcb6bf28>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_threshold : <cocotb_coverage.coverage.CoverPoint object at 0x2aaedcb6be80>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.fifo_underflow : <cocotb_coverage.coverage.CoverPoint object at 0x2aaedcd6c048>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 10
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rw : <cocotb_coverage.coverage.CoverPoint object at 0x2aaedcb6be10>, coverage=1, size=2
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN True : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN False : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXempty : <cocotb_coverage.coverage.CoverCross object at 0x2aaedcd6c080>, coverage=1, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXfull : <cocotb_coverage.coverage.CoverCross object at 0x2aaedcd6c160>, coverage=1, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXoverflow : <cocotb_coverage.coverage.CoverCross object at 0x2aaedcd6c198>, coverage=1, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXthreshold : <cocotb_coverage.coverage.CoverCross object at 0x2aaedcd6c128>, coverage=1, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 0
3.95ns INFO cocotb.test coverage.py:95 in report_coverage top.rwXunderflow : <cocotb_coverage.coverage.CoverCross object at 0x2aaedcd6c1d0>, coverage=1, size=4
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (True, False) : 10
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, True) : 0
3.95ns INFO cocotb.test coverage.py:100 in report_coverage BIN (False, False) : 0
3.95ns INFO cocotb.regression regression.py:351 in _score_test Test Passed: fifo_test
3.95ns INFO cocotb.regression regression.py:469 in _log_test_summary Passed 1 tests (0 skipped)
3.95ns INFO cocotb.regression regression.py:538 in _log_test_summary *****************************************************************************
** TEST PASS/FAIL SIM TIME(NS) REAL TIME(S) RATIO(NS/S) **
*****************************************************************************
** test_fifo.fifo_test PASS 3.95 0.08 49.36 **
*****************************************************************************
3.95ns INFO cocotb.regression regression.py:555 in _log_sim_summary *************************************************************************************
** ERRORS : 0 **
*************************************************************************************
** SIM TIME : 3.95 NS **
** REAL TIME : 0.10 S **
** SIM / REAL TIME : 39.39 NS/S **
*************************************************************************************
3.95ns INFO cocotb.regression regression.py:249 in tear_down Shutting down...
Package versions as below:
cocotb 1.4.0.dev0
cocotb-coverage 1.0.0
Many thanks for your help!
Question about Packet Switch Example
Hi, I have run this example successfully and almost understand every part of the codes. But there is two points that I can't figure out.
First: Why coverpoint top.filt_addr only have 32 possible options?
Since addr & mask has 8 binary bits, it should be 256 possible options. Also, I found that was't every packet operation output the top.filt_addr bin. There must be some point I missed. I guess this coverpoint bin only activate under some condition and so there is only 32 possible options, but I can't figure out.
@CoverPoint(
"top.filt_addr",
xf = lambda pkt, event, addr, mask, ll, ul: addr & mask, #filtering based on a particular bits in header
bins = list(range(32)) #all options possible
)
Second: I was wondering why we limited low_limit value in 3 ~ 30. Does it mean anything wrong when low_limit = 31 ?
On the other hand, range(3, 31) has values from 3 ~ 30, but random.randint(3,31) has values from 3 ~ 31. Am I correct?
@CoverPoint(
"top.filt_len_ll",
vname = "ll", #lower limit of packet length
bins = list(range(3,31))
)
@CoverPoint(
"top.filt_len_ul",
vname = "ll", #upper limit of packet length
bins = list(range(3,32))
)
low_limit = random.randint(3,31)
up_limit = random.randint(low_limit,32)
Many thanks for your contributions and helps!
Typos in documentation
Documentation reference.html https://cocotb-coverage.readthedocs.io/en/latest/reference.html CoverPoint example:
>>> @coverage.CoverPoint( # cover (arg/2) < 1...5 (5 bins)
...
bins = list(range(1, 5))should be 4 bins
CoverCross example:
4 bins total, 4x4 - 2 = 14 bins total
https://cocotb-coverage.readthedocs.io/en/latest/examples.html
Testbench section
bins = list(range(3,32)) #may be 3 ... 32 bytes
Should be 31 bytes, range() is not inclusive
And below that in the TB code:
low_limit = random.randint(3,31)
up_limit = random.randint(low_limit,32)Here I would suggest using randrange(3, 32) and randrange(low_limit, 33) because it is consistent with range() and how indexing works in python and the rest of documentation.
Problems with tutorial transitions example
The documentation tutorial page section about transitions has several issues:
- bins as list-of-list does not work, throws
TypeError: unhashable type: 'list'(works if changed to list-of-tuple) - appending to
addr_previn therelfunction causes the sample to be appended multiple times, once for each bin. The append needs to be moved to a separatexfto only happen once - transition
0 1 2 3bin cannot be hit as there is a0 1bin appearing before it, andinjis not set.
Also, inj does not work as documented, I had to explicitly set it to False to get this example to behave. See #63
Using randomize() or randomize_with() in list comprehension doesn't work
Any idea why I can't generate a list of randomized transactions using the following?
from cocotb_coverage.crv import Randomized
class Test(Randomized):
def __init__(self):
Randomized.__init__(self)
self.x = 0
self.y = 0
self.add_rand("x", list(range(10)))
self.add_rand("y", list(range(20)))
def c_constraint(x):
return x == 4
self.add_constraint(c_constraint)
def __repr__(self):
return f"x={self.x} y={self.y}"
if __name__ == "__main__":
print("this works")
tests = []
for _ in range(10):
t = Test()
t.randomize_with(lambda x: 1 <= x <= 4)
tests.append(t)
print(tests)
print("\nthis doesn't")
tests = [Test().randomize_with(lambda x: 1 <= x <= 4) for _ in range(10)]
print(tests)
print("\nnor this")
tests = [t.randomize_with(lambda x: 1 <= x <= 4) for t in [Test() for _ in range(10)]]
print(tests)results in:
this works
[x=1 y=15, x=1 y=19, x=4 y=4, x=3 y=4, x=4 y=5, x=4 y=16, x=1 y=19, x=2 y=19, x=1 y=18, x=2 y=6]
this doesn't
[None, None, None, None, None, None, None, None, None, None]
nor this
[None, None, None, None, None, None, None, None, None, None]
Missing "weight" variable in CoverItem class + make "weight" accessible using @property
CoverPoint, CoverCross and CoverItem classes should have a retrievable weight property.
CoverItem instance weight should be a sum of his children weights.
randomize_with() fails
Hi,
I've got the following Randomized class with two variables, dac_max and dac_min, that I want to constrain. The class definition looks like:
self.add_rand("dac_max", list(range(512)))
self.add_rand("dac_min", list(range(512)))
self.add_constraint(lambda dac_max, dac_min : dac_max > dac_min )
and I'm able to add further constraints without issue. Such as:
self.add_constraint(lambda dac_max : dac_max == 511 )
However, if I try to add a constraint with a call to randomize_with(), I get an exception raised. What is wrong with the following usage of randomize_with()?
tx.randomize_with(lambda dac_max : dac_max == 384, lambda dac_min : dac_min == 32)
Shareef.
`CoverCross` doesn't update its `new_hits` field
CoverCross objects don't update their _new_hits list. So this info can't be obtained addressing cover_cross_object.new_hits field.
Use iterable range without conversion to list as domain in add_rand examples
The add_rand method of the crv.Randomized class expects an iterable domain. All given examples transform iterable ranges to lists, e.g.:
def Point(crv.Randomized):
def __init__(self, x, y):
# ...
self.add_rand('x', list(range(10)))
Why is this transformation necessary? It is not scalable to large ranges, because Python will construct the whole list in memory. The use of domain within constraint resolving appears to be only iteration. This works on ranges even more efficiently:
$ python3 -m timeit 'for i in range(0xFFFF): pass'
50 loops, best of 5: 8.57 msec per loop
$ python3 -m timeit 'for i in list(range(0xFFFF)): pass'
20 loops, best of 5: 11 msec per loop
solve_order issue
I have the following test code. For some reason, if I specify the solver_order statement, it will always give me an error:
class my_random(Randomized):
def __init__(self):
super().__init__()
self.x = 0
self.y = 0
self.z = 0
self.add_rand('x', list(range(10)))
self.add_rand('y', list(range(10)))
self.add_rand('z', list(range(10)))
self.add_constraint(lambda x: 0 <= x <= 5)
self.add_constraint(lambda y: 0 <= y <= 6)
self.add_constraint(lambda x,y,z: x+y+z < 15)
self.solve_order(['x', 'y'], 'z')
def print_vars(self):
print("x={}, y={}, z={}, sum={}".format(self.x, self.y, self.z, self.x+self.y+self.z))
if __name__ == '__main__':
my_obj = my_random()
my_obj.add_constraint(lambda x,y,z: x+y+z < 10)
my_obj.randomize()
my_obj.print_vars()
The error is like this:
File "/prj/tbs/5g/sandiego/hw/scratch/sims/hctseng/pandace3/venv3.5.2/lib/python3.5/site-packages/cocotb_coverage/crv.py", line 401, in _del_constraint
assert(0), "Could not delete a constraint!"
AssertionError: Could not delete a constraint!
If I change the solve_order() in this way, it will give me the same error:
self.solve_order('x', 'y', 'z')
Other questions:
If I have multiple solve_order() statements, will it honor all of them, or just the last statement?
What is the definition of hard and soft constraint in COCOTB?
Many thanks!
Richard
Documentation bug with lambdas
I think there are bugs in the example code in the introduction:
xf = lambda xfer.length,
lambda requires a colon after the keyword even if it takes no parameters.
How to report merged coverage?
Related to #22,
Each of my testcase exports its coverage data in an XML file.
At the end of the regression I call a script which finds all 'coverage.xml' file and calls
merge_coverage(log.info, "functional_coverage.xml", *all_files)
Now the next step is to call
{some coverage_db}.report_coverage()
Looking at the code it looks like the merged_db is local to the function and cannot be used.
Is there some other way to report the merged data?
Access to `coverage_db['top'].detailed_coverage` field is failed.
CoverItem class contains detailed_coverage property and it can be accessed accordingly to spec. But trying to address it (for instance: coverage_db['top'].detailed_coverage) leads to fail. The cause of exception is using append() method for a dict in CoverItem code:
coverage = {}
for child in self._children:
coverage.append(child.detailed_coverage)
support python2
To support python2, a file name with init.py must create under the package folder.
touch cocotb_coverage/init.py
Thanks.
https://stackoverflow.com/questions/37139786/is-init-py-not-required-for-packages-in-python-3
Version 1.1.dev in pypi
Is this 1.1.dev version available from https://pypi.org/project/cocotb-coverage? If not, can you please add in there? The issue is the 1.0.0 version does not work with the solve_order correctly. I have to use 1.1.dev, but our IT support said it will be very difficult to pull in the open source module that is not from pypi.org.
Thanks,
Richard
Error in merging two or more files
Following error is occurs when merging two or more files with function
cov.merge_coverage(print, "cov.xml", "cov_1.xml", "cov_2.xml")
Error:
~/.conda/envs/p_env/lib/python3.7/site-packages/cocotb_coverage/coverage.py in merge_element(db)
1082 parent_hits_threshold = int(
1083 name_to_elem[parent_name].attrib['at_least'])
-> 1084 if (hits_orig < parent_hits_threshold
1085 and hits_orig+hits >= parent_hits_threshold):
1086 update_parent(name=abs_name, bin_update=True,
UnboundLocalError: local variable 'hits_orig' referenced before assignment
camelCase method names
Hi,
for instance reportCoverage and coverageSection are camelCased - is there a reason why they are not lowercase with underscores like the rest?
Run on Azure with coverage report ?
Would it work if we setup CI with Azure pipelines and report coverage directly like here: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/review-code-coverage-results?view=azure-devops
I can try to set this up.
constrained random and functional coverage
hi,
i have to randomise some bits of 32 bit instruction
and some bits will be constant like below format
+---------+-------+-------+-----+-------+---------+
| 0100000 | XXXXX | XXXXX | 111 | XXXXX | 0110011 |
+---------+-------+-------+-----+-------+---------+
through direct random function we can randomise cant find coverage
using cocotb-coverage library.
for i in range(n):
instruction = random.randint(0,0xFFFFFFFF)
instruction = instruction & 0x01FF8F80
instruction = instruction | 0x40007033
how we can find constrained random and coverage using cocotb-coverage?
Ignore whitespace in signal names
Hi,
I just found a very obscure issue caused by me specifying the add_rand() method with whitespace at the end of the signal names. I was using rectangle edit in emacs to insert a bunch of long names and ended up with something like:
self.add_rand("sig_name_a ", list(range(10)))
self.add_rand("sig_name_ab", list(range(10)))
which when you run a test, ends up with 0 always assigned to the signals with whitespace. In the example above, sig_name_a is never randomized.
Could you fire a warning/error for this or just strip off any whitespace?
Cheers.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.