freechipsproject / chisel-testers Goto Github PK

View Code? Open in Web Editor NEW

98.0 98.0 50.0 1.22 MB

Provides various testers for chisel users

License: Apache License 2.0

Scala 92.79% C++ 6.83% Verilog 0.07% Makefile 0.31%

chisel-testers's People

Contributors

Stargazers

Watchers

chisel-testers's Issues

TestApplicationException with 0 exit code is suppressed

PeekPokeTester catches TestApplicationException, returning failure if the exit code is non-zero. Some simulations (notably vcs) do not set a reliable exit code, but in any case, we should signal up-level that the simulation is no longer running.

Should the verilator backend let you peek/poke an analogType?

If, in certain conditions, it's not allowed, I wouldn't mind having Verilator fail... but otherwise, even if an inout port (of a blackbox) is actually just the sume of two inputs, I can't actually probe that it's right. Basically, I want to sanity check that connections are made correctly, and I can't do that in simulation.

Patch needed when update to scalatest 3.0.1

I am trying to get the IntelliJ debugger working with Chisel and Chisel-Testers. The issue I am facing is a ClassNotFoundException for: org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner

I root caused (I believe) to a dependency on newer scalatest version by IDEA Scala plugin and chisel-testers.

Uprev chisel-testers to latest mater and scalatest version to 3.0.1 results in compile error. Output for sbt compile follows:

[error] /home/edc/chisel_plus/chisel-testers/src/main/scala/chisel3/iotesters/ChiselSpec.scala:32: overriding value generatorDrivenConfig in trait Configuration of type ChiselPropSpec.this.PropertyCheckConfiguration;
[error]  value generatorDrivenConfig has incompatible type
[error]   implicit override val generatorDrivenConfig =
[error]                         ^

[error] one error found

Once time permits, I'll continue to workaround, but until then, can developers take a look?

PeekPokeTester: including simulation time assert (or stop) causes verilator compilation error

We're trying to place runtime asserts in mocked implementation code. The following idea works using firrtl-interpreter and SteppedHWIOTester, but not with the PeekPokeTester due to a missing symbol.

g++    myRisc-harness.o verilated.o verilated_vcd_c.o VmyRisc__ALL.a    -o VmyRisc -lm -lstdc++  2>&1 | c++filt
VmyRisc__ALL.a(VmyRisc__ALLcls.o): In function `VmyRisc::_sequent__TOP__3(VmyRisc__Syms*)':
VmyRisc__ALLcls.cpp:(.text+0x1ca): undefined reference to `sc_time_stamp()'
collect2: error: ld returned 1 exit status

Here is the source code and the complete log file.

package myRisc

import chisel3._

class mockRegFileOut() extends Module{
  val io = new Bundle {
    val addr = Bits(INPUT, 8)
    val DataO = Bits(OUTPUT, 32)
  }

  assert(io.addr === UInt(0),"ASSERT:Read at the wrong Register")

  when(io.addr === UInt(0)){
    io.DataO := UInt(128)
  }
}

class myRisc extends Module{
  val io = new Bundle {
    val wrData = Bits(INPUT, 32)

    val valid  = Bool(OUTPUT)
    val out    = Bits(OUTPUT, 32)
  }

  val inst = io.wrData


  //instantiate mock
  val mock = Module(new mockRegFileOut).io
  mock.addr := UInt(16)//inst(15, 8)
// Comment the next line to see a mocking failure
  mock.addr := inst(15, 8)


  val rci  = inst(23,16) 

  when (rci === UInt(255)){
    io.valid := Bool(true)
    io.out := mock.DataO
  }.otherwise{
    io.valid := Bool(false)
  }

}

import org.scalatest.{ Matchers, FlatSpec, GivenWhenThen}

import chisel3.iotesters._

class myRiscPeekPokeTester( val c:myRisc) extends PeekPokeTester(c) {
   poke( c.io.wrData, (0<<24)+(255<<16))
   step(1)
   expect( c.io.valid, 1)
   expect( c.io.out, 128)
}

class myRiscPeekPokeTest extends FlatSpec with Matchers {
  behavior of "myRiscPeekPokeTest"

  it should s"work" in {
    chisel3.iotesters.Driver((() => new myRisc),"verilator") { c => new myRiscPeekPokeTester(c) } should be (true)
  }  
}

The output of "sbt test":

> test
[info] [0.002] Elaborating design...
[info] [0.073] Done elaborating.
verilator --cc myRisc.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY --trace -O2 --top-module myRisc +define+TOP_TYPE=VmyRisc +define+PRINTF_COND=!myRisc.reset +define+STOP_COND=!myRisc.reset -CFLAGS -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VmyRisc -include VmyRisc.h -Mdir test_run_dir/myRisc.myRisc --exe myRisc-harness.cpp
make: Entering directory `/nfs/site/disks/scl.work.59/ppt/smburns/tdd_using_chisel-testing_flows/bugs/bug-sc_time_stamp/test_run_dir/myRisc.myRisc'
g++  -I.  -MMD -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VmyRisc -include VmyRisc.h   -c -o myRisc-harness.o myRisc-harness.cpp
g++  -I.  -MMD -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VmyRisc -include VmyRisc.h   -c -o verilated.o /nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/verilated.cpp
g++  -I.  -MMD -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VmyRisc -include VmyRisc.h   -c -o verilated_vcd_c.o /nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/verilated_vcd_c.cpp
/usr/intel/bin/perl /nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include VmyRisc.cpp > VmyRisc__ALLcls.cpp
/usr/intel/bin/perl /nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include VmyRisc__Trace.cpp VmyRisc__Syms.cpp VmyRisc__Trace__Slow.cpp > VmyRisc__ALLsup.cpp
g++  -I.  -MMD -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VmyRisc -include VmyRisc.h   -c -o VmyRisc__ALLsup.o VmyRisc__ALLsup.cpp
g++  -I.  -MMD -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include -I/nfs/site/disks/scl.work.59/ppt/smburns/chisel/chisel3/verilator_install/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VmyRisc -include VmyRisc.h   -c -o VmyRisc__ALLcls.o VmyRisc__ALLcls.cpp
      Archiving VmyRisc__ALL.a ...
ar r VmyRisc__ALL.a VmyRisc__ALLcls.o VmyRisc__ALLsup.o
ranlib VmyRisc__ALL.a
g++    myRisc-harness.o verilated.o verilated_vcd_c.o VmyRisc__ALL.a    -o VmyRisc -lm -lstdc++  2>&1 | c++filt
VmyRisc__ALL.a(VmyRisc__ALLcls.o): In function `VmyRisc::_sequent__TOP__3(VmyRisc__Syms*)':
VmyRisc__ALLcls.cpp:(.text+0x1ca): undefined reference to `sc_time_stamp()'
collect2: error: ld returned 1 exit status
make: Leaving directory `/nfs/site/disks/scl.work.59/ppt/smburns/tdd_using_chisel-testing_flows/bugs/bug-sc_time_stamp/test_run_dir/myRisc.myRisc'
[info] myRiscPeekPokeTest:
[info] myRiscPeekPokeTest
[info] - should work *** FAILED ***
[info]   java.lang.IllegalArgumentException: requirement failed: test_run_dir/myRisc.myRisc/VmyRisc doesn't exists
[info]   at scala.Predef$.require(Predef.scala:219)
[info]   at chisel3.iotesters.TesterProcess$.apply(PeekPokeTesterUtils.scala:112)
[info]   at chisel3.iotesters.SimApiInterface.<init>(SimApiInterface.scala:278)
[info]   at chisel3.iotesters.VerilatorBackend.<init>(VerilatorBackend.scala:290)
[info]   at chisel3.iotesters.setupVerilatorBackend$.apply(VerilatorBackend.scala:282)
[info]   at chisel3.iotesters.Driver$.apply(Driver.scala:22)
[info]   at myRisc.myRiscPeekPokeTest$$anonfun$4.apply$mcV$sp(t.scala:62)
[info]   at myRisc.myRiscPeekPokeTest$$anonfun$4.apply(t.scala:62)
[info]   at myRisc.myRiscPeekPokeTest$$anonfun$4.apply(t.scala:62)
[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info]   ...
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] ScalaTest
[info] Run completed in 3 seconds, 1 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error]         myRisc.myRiscPeekPokeTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 3 s, completed Sep 21, 2016 11:50:06 AM

AdvTester: Second of two tests fails when run together---each works when run individually

I've tried to do some parameterized design in chisel3.
I've defined two tests which use testbenches and implemention of the same module but with different parameters. It seems that the implementation in the second test is not elaborated (caused the implementation with different/wrong parameters to be used.)

Here is a complete file that will expose this issue.

package bug

import chisel3._
import chisel3.iotesters._
import org.scalatest.GivenWhenThen
import scala.runtime.ScalaRunTime.stringOf

class CarryChainIfc(N:Int,width:Int) extends Module {
   val io = new Bundle {
      val a  = Vec( N, UInt(INPUT,width=width))
      val co = Vec( N, UInt(OUTPUT,width=width))
   }
}

class ORChain(N:Int) extends CarryChainIfc(N,1) {
   require( io.a.length == io.co.length)
   require( io.a.length == N)
   io.co(0) := io.a(0)
   for( ((a,ci),co)<-io.a.toList.tail.zip( io.co).zip( io.co.tail)) {
      co := a | ci
   }
}

abstract class Action
case class ActionInputArray(val nm:String, val s:Array[BigInt])  extends Action
case class ActionOutputArray(val nm:String, val s:Array[BigInt])  extends Action
case class ActionDelay(val v:Int)  extends Action

class CarryChainTests[T <: CarryChainIfc]( ActList:List[Action], c: T) extends AdvTester(c,verbose=true)
{

  for (a <- ActList) {  
    a match {
    case ActionInputArray("a",s) => {
       require( s.length == c.io.a.length)
       for ( (f,a) <- c.io.a.zip( s)) {
          reg_poke( f, a)
       }
    }
    case ActionOutputArray("co",s) => {
       require( s.length == c.io.co.length)
       for ( (f,a) <- c.io.co.zip( s)) {
          expect( f, a)
       }
    }
    case ActionDelay(v) => takesteps(1)()
    }
  }
}

class CarryChainTester[ +T <: CarryChainIfc]( val dut: () => T) extends ChiselFlatSpec with GivenWhenThen {
//  val backend = "firrtl"
  val backend = "verilator"
  val action_buffer= new collection.mutable.ListBuffer[Action]()

  def InputArray( nm:String, tag:String, s:Array[BigInt]):String = {
     action_buffer += new ActionInputArray(nm,s)
     s"InputArray $nm $tag ${stringOf(s)}"
  }

  def OutputArray( nm:String, tag:String, s:Array[BigInt]):String = {
     action_buffer += new ActionOutputArray(nm,s)
     s"OutputArray $nm $tag ${stringOf(s)}"
  }

  def Delay(v:Int):String = {
     action_buffer += new ActionDelay(v)
     s"After $v cycle" + (if ( v != 1) "s" else "")
  }
  def init():Unit = action_buffer.clear
  def run():Boolean = {
     chisel3.iotesters.Driver(dut,backend) {
       (component) =>
           new CarryChainTests( action_buffer.toList,component)
     }
  }
}

class ORChainTester(val N:Int) extends CarryChainTester( ()=>new ORChain(N)) {
  def v(bin: String): Array[BigInt] = {
     assert( N === bin.length)
     bin.toList.map( "01".indexOf(_)).map( BigInt(_)).reverse.toArray
  }
}


class ORChainTest2 extends ORChainTester(2) {
  s"ORChain of size ${N}" should s"work with various inputs" in {
    init()
    Given("a ORChain module")
    for ( (a,co) <- List( (v("00"),v("00")),
                          (v("01"),v("11")),
                          (v("10"),v("10")),
                          (v("11"),v("11")))) {
      When(InputArray("a","is",a))
      And(Delay(1))
      Then(OutputArray("co","is",co))
    }
    run() should be (true)
  }
}

class ORChainTest4 extends ORChainTester(4) {
  s"ORChain of size ${N}" should s"work with various inputs" in {
    init()
    Given("a ORChain module")
    for ( (a,co) <- List( (v("0000"),v("0000")),
                          (v("0001"),v("1111")),
                          (v("0010"),v("1110")),
                          (v("0100"),v("1100")),
                          (v("1000"),v("1000")))) {
      When(InputArray("a","is",a))
      And(Delay(1))
      Then(OutputArray("co","is",co))
    }
    run() should be (true)
  }
}

I get this LOG file with a compiler error when the second module is compiled (it then continues to run the wrong test.)

smburns@smburns-VirtualBox:~/bug-parameterized-dut$ sbt test
[info] Set current project to SCL PPT Testing Experiments (in build file:/home/smburns/bug-parameterized-dut/)
[info] [0.004] Elaborating design...
[info] [0.287] Done elaborating.
verilator --cc ORChain.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY --trace -O2 --top-module ORChain +define+TOP_TYPE=VORChain +define+PRINTF_COND=!ORChain.reset +define+STOP_COND=!ORChain.reset -CFLAGS -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h -Mdir test_run_dir/bug.ORChain --exe ORChain-harness.cpp
make: Entering directory `/home/smburns/bug-parameterized-dut/test_run_dir/bug.ORChain'
g++  -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h   -c -o ORChain-harness.o ORChain-harness.cpp
g++  -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h   -c -o verilated.o /usr/local/share/verilator/include/verilated.cpp
g++  -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h   -c -o verilated_vcd_c.o /usr/local/share/verilator/include/verilated_vcd_c.cpp
/usr/bin/perl /usr/local/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include VORChain.cpp > VORChain__ALLcls.cpp
/usr/bin/perl /usr/local/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include VORChain__Trace.cpp VORChain__Syms.cpp VORChain__Trace__Slow.cpp > VORChain__ALLsup.cpp
g++  -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h   -c -o VORChain__ALLcls.o VORChain__ALLcls.cpp
g++  -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h   -c -o VORChain__ALLsup.o VORChain__ALLsup.cpp
      Archiving VORChain__ALL.a ...
ar r VORChain__ALL.a VORChain__ALLcls.o VORChain__ALLsup.o
ar: creating VORChain__ALL.a
ranlib VORChain__ALL.a
g++    ORChain-harness.o verilated.o verilated_vcd_c.o VORChain__ALL.a    -o VORChain -lm -lstdc++  2>&1 | c++filt
make: Leaving directory `/home/smburns/bug-parameterized-dut/test_run_dir/bug.ORChain'
sim start on smburns-VirtualBox at Fri Sep 16 15:03:52 2016
inChannelName: 00009370.in
outChannelName: 00009370.out
cmdChannelName: 00009370.cmd
STARTING test_run_dir/bug.ORChain/VORChain
SEED 1474063432674
STEP 0 -> 1
  POKE ORChain.io_a_1 <- 0x0
  POKE ORChain.io_a_0 <- 0x0
  EXPECT ORChain.io_co_0 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_1 -> 0x0 == 0x0 PASS
STEP 1 -> 2
  POKE ORChain.io_a_1 <- 0x0
  POKE ORChain.io_a_0 <- 0x1
  EXPECT ORChain.io_co_0 -> 0x1 == 0x1 PASS
  EXPECT ORChain.io_co_1 -> 0x1 == 0x1 PASS
STEP 2 -> 3
  POKE ORChain.io_a_1 <- 0x1
  POKE ORChain.io_a_0 <- 0x0
  EXPECT ORChain.io_co_0 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_1 -> 0x1 == 0x1 PASS
STEP 3 -> 4
  POKE ORChain.io_a_1 <- 0x1
  POKE ORChain.io_a_0 <- 0x1
  EXPECT ORChain.io_co_0 -> 0x1 == 0x1 PASS
  EXPECT ORChain.io_co_1 -> 0x1 == 0x1 PASS
Enabling waves..
RAN 4 CYCLES PASSED
[info] ORChainTest2:
[info] ORChain of size 2
[info] - should work with various inputs
[info]   + Given a ORChain module 
[info]   + When InputArray a is Array(0, 0) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(0, 0) 
[info]   + When InputArray a is Array(1, 0) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(1, 1) 
[info]   + When InputArray a is Array(0, 1) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(0, 1) 
[info]   + When InputArray a is Array(1, 1) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(1, 1) 
[info] [0.000] Elaborating design...
[info] [0.017] Done elaborating.
verilator --cc ORChain.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY --trace -O2 --top-module ORChain +define+TOP_TYPE=VORChain +define+PRINTF_COND=!ORChain.reset +define+STOP_COND=!ORChain.reset -CFLAGS -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h -Mdir test_run_dir/bug.ORChain --exe ORChain-harness.cpp
make: Entering directory `/home/smburns/bug-parameterized-dut/test_run_dir/bug.ORChain'
g++  -I.  -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O2 -DTOP_TYPE=VORChain -include VORChain.h   -c -o ORChain-harness.o ORChain-harness.cpp
ORChain-harness.cpp: In member function ‘void ORChain_api_t::init_sim_data()’:
ORChain-harness.cpp:24:61: error: ‘class VORChain’ has no member named ‘io_a_2’
         sim_data.inputs.push_back(new VerilatorCData(&(dut->io_a_2)));
                                                             ^
ORChain-harness.cpp:25:61: error: ‘class VORChain’ has no member named ‘io_a_3’
         sim_data.inputs.push_back(new VerilatorCData(&(dut->io_a_3)));
                                                             ^
ORChain-harness.cpp:28:62: error: ‘class VORChain’ has no member named ‘io_co_2’
         sim_data.outputs.push_back(new VerilatorCData(&(dut->io_co_2)));
                                                              ^
ORChain-harness.cpp:29:62: error: ‘class VORChain’ has no member named ‘io_co_3’
         sim_data.outputs.push_back(new VerilatorCData(&(dut->io_co_3)));
                                                              ^
At global scope:
cc1plus: warning: unrecognized command line option "-Wno-undefined-bool-conversion" [enabled by default]
cc1plus: warning: unrecognized command line option "-Wno-parentheses-equality" [enabled by default]
make: *** [ORChain-harness.o] Error 1
make: Leaving directory `/home/smburns/bug-parameterized-dut/test_run_dir/bug.ORChain'
sim start on smburns-VirtualBox at Fri Sep 16 15:03:54 2016
inChannelName: 00009388.in
outChannelName: 00009388.out
cmdChannelName: 00009388.cmd
STARTING test_run_dir/bug.ORChain/VORChain
SEED 1474063434297
STEP 0 -> 1
  POKE ORChain.io_a_1 <- 0x0
  POKE ORChain.io_a_0 <- 0x0
  POKE ORChain.io_a_3 <- 0x0
  POKE ORChain.io_a_2 <- 0x0
  EXPECT ORChain.io_co_0 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_1 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_2 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_3 -> 0x0 == 0x0 PASS
STEP 1 -> 2
  POKE ORChain.io_a_1 <- 0x0
  POKE ORChain.io_a_0 <- 0x1
  POKE ORChain.io_a_3 <- 0x0
  POKE ORChain.io_a_2 <- 0x0
  EXPECT ORChain.io_co_0 -> 0x1 == 0x1 PASS
  EXPECT ORChain.io_co_1 -> 0x1 == 0x1 PASS
  EXPECT ORChain.io_co_2 -> 0x0 == 0x1 FAIL
  EXPECT ORChain.io_co_3 -> 0x0 == 0x1 FAIL
STEP 2 -> 3
  POKE ORChain.io_a_1 <- 0x1
  POKE ORChain.io_a_0 <- 0x0
  POKE ORChain.io_a_3 <- 0x0
  POKE ORChain.io_a_2 <- 0x0
  EXPECT ORChain.io_co_0 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_1 -> 0x1 == 0x1 PASS
  EXPECT ORChain.io_co_2 -> 0x0 == 0x1 FAIL
  EXPECT ORChain.io_co_3 -> 0x0 == 0x1 FAIL
STEP 3 -> 4
  POKE ORChain.io_a_1 <- 0x0
  POKE ORChain.io_a_0 <- 0x0
  POKE ORChain.io_a_3 <- 0x0
  POKE ORChain.io_a_2 <- 0x1
  EXPECT ORChain.io_co_0 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_1 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_2 -> 0x0 == 0x1 FAIL
  EXPECT ORChain.io_co_3 -> 0x0 == 0x1 FAIL
STEP 4 -> 5
  POKE ORChain.io_a_1 <- 0x0
  POKE ORChain.io_a_0 <- 0x0
  POKE ORChain.io_a_3 <- 0x1
  POKE ORChain.io_a_2 <- 0x0
  EXPECT ORChain.io_co_0 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_1 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_2 -> 0x0 == 0x0 PASS
  EXPECT ORChain.io_co_3 -> 0x0 == 0x1 FAIL
Enabling waves..
RAN 5 CYCLES FAILED FIRST AT CYCLE 2
[info] ORChainTest4:
[info] ORChain of size 4
[info] - should work with various inputs *** FAILED ***
[info]   false was not true (bug.scala:116)
[info]   + Given a ORChain module 
[info]   + When InputArray a is Array(0, 0, 0, 0) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(0, 0, 0, 0) 
[info]   + When InputArray a is Array(1, 0, 0, 0) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(1, 1, 1, 1) 
[info]   + When InputArray a is Array(0, 1, 0, 0) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(0, 1, 1, 1) 
[info]   + When InputArray a is Array(0, 0, 1, 0) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(0, 0, 1, 1) 
[info]   + When InputArray a is Array(0, 0, 0, 1) 
[info]   + And After 1 cycle 
[info]   + Then OutputArray co is Array(0, 0, 0, 1) 
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] ScalaTest
[info] Run completed in 11 seconds, 89 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 2, Failed 1, Errors 0, Passed 1
[error] Failed tests:
[error]     bug.ORChainTest4
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 14 s, completed Sep 16, 2016 3:03:55 PM

Ability to add to command line for verilator and vcs backends

Asked on stackoverflow.

Is there a way to specify options to VCS when using the PeekPokeTester? In particular, I would like to: 1) Enable System Verilog for black box code 2) Control the macro defines to disable the random number-based initialization. (I want to see initialization issues as Xs.)

I'd propose that there be two new options to chisel-testers, one to specify additional command line arguments, and another to expand the subcommands that are passed to the C compiler.

PeekPokeTester.peekAt/pokeAt broken?

I've declared a memory via val mem = Mem(sz, UInt(64.W)), but peekAt just returns random values, even when using pokeAt on the same addresses a few cycles earlier. Is this bug, or am I missing something here? (I'm aware that peek and poke now only work on values in the top-level IO; but since Mems cannot be declared in the Bundle, how can they be accessed efficiently in simulation?)

Backends and their setup routines do not use Driver.execute family properly

For now, the backends for verilator and vcs do not use the latest Driver.execute api methods.
This should be fixed.

Reduce the startup time for a Scala Unit-test by skipping Verilator compilation and design elaboration

To interface to an existing test-system, I would like to launch a test from the command line hundreds of times by passing different test-input vector files on the command line.

E.g. given the input data in the "test.txt" file, I'd like to execute a test given:

$ sbt "test:runMain testmain test.txt"

The code below skips Verilator compilation(which is slow), but still performs design elaboration:

    Driver.run(() => new system(),
        "Vsystem")(c =>
          new system_Tests(c, args(0)))

assert in AdvTester does not print error message

The code seems to just ignore the second argument to assert.

random seed handling is not clean

Make it clean. Should be set in one consistent way

Get rid of the test boilerplate

This is what we have right now

class MyDUT extends Module {
  ... instantiate MyDevice here ...
}

class MyDUTTester(val c: MyDUT) extends PeekPokeTester(c) {
  expect(c.io.blah, someNumber)
  ...
}

class MyDeviceSpec extends ChiselFlatSpec {
  "MyDevice" should "blah" in {
    Driver(() => new MyDUT) {
      c => new MyDUTTester(c)
    } should be (true)
  }
}

The amount of repetition there is pretty insane, and even worse is that the Tester logic is its own class, but is dependent on the parameterization that is specified externally in the Spec. Individual expect failures also aren't registered by the test framework (which can automatically generate nice error messages at the invocation site, I think including a stack trace); instead the test framework only tells your test failed somewhere. (yes, the default verbose failures messages helps, but that's a workaround at best, and for whatever reason isn't showing up on Verilator tests)

I think ideally we would have something like:

class MyDeviceSpec extends ChiselPeekPokeSpec {
  "MyDevice" should "blah" in {
    val t, c = createTester(new MyDUT)  // essentially creates a test fixture, return an emulator link and the elaborated module for IO access
    t.expect(c.io.blah, someNumber)
   ...
  }
}

Much more compact, less repetition, less boilerplate, and most importantly makes writing tests fun for everyone!

Thoughts?

New option "--no-check-comb-loops" doesn't work with PeekPokeTester (possibly others)

The new option to "firrtl" isn't enabled in the testing flow.
This code still flags a combinational loop. (Does work from "firrtl" (command line) and the chisel3.Driver methods.)

package bug

import org.scalatest.{ Matchers, FlatSpec, GivenWhenThen}

import chisel3._
import chisel3.util._
import chisel3.iotesters._

class HasCycle extends Module {
  val io = IO( new Bundle {
    val a = Input(Bool())
    val o = Output(Bool())
  })

  val b = Wire(Bool())
  b := b&&io.a

  io.o := b
}

class HasCycleTester( c:HasCycle) extends PeekPokeTester(c) {
  poke( c.io.a, 0)
  step(1)
}

class HasCycleTest extends FlatSpec with Matchers {
  behavior of "HasCycle"
  it should "work" in {
    chisel3.iotesters.Driver.execute( Array( "--no-check-comb-loops", "--backend-name", "verilator"), () => new HasCycle) { c =>
      new HasCycleTester( c)
    } should be ( true)
  }
}

Here is the error:

[info] HasCycleTest:
[info] HasCycle
[info] [0.003] Elaborating design...
[info] [0.149] Done elaborating.
[info] - should work *** FAILED ***
[info]   firrtl.transforms.CheckCombLoops$CombLoopException: : [module HasCycle] Combinational loop detected:
[info] HasCycle._T_5
[info] HasCycle.b
[info] HasCycle._T_5
[info]   at firrtl.transforms.CheckCombLoops$$anonfun$run$1$$anonfun$apply$8.apply(CheckCombLoops.scala:216)
[info]   at firrtl.transforms.CheckCombLoops$$anonfun$run$1$$anonfun$apply$8.apply(CheckCombLoops.scala:211)
[info]   at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[info]   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
[info]   at firrtl.transforms.CheckCombLoops$$anonfun$run$1.apply(CheckCombLoops.scala:211)
[info]   at firrtl.transforms.CheckCombLoops$$anonfun$run$1.apply(CheckCombLoops.scala:204)
[info]   at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[info]   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
[info]   at firrtl.transforms.CheckCombLoops.run(CheckCombLoops.scala:204)
[info]   at firrtl.transforms.CheckCombLoops.execute(CheckCombLoops.scala:231)
[info]   ...
[info] ScalaTest

Package Name?

Is there any problem with Chisel.testers._ ?

Not specifying direction of inputs, could be detected by tester

There are situation when the directions of io are not specified that the tester does not detect any of the inputs. I think the tester should throw consider throwing an exception when no inputs are detected.
Alternatively, the tester should assume that unspecfied directions are outputs, which seems to be the current assumption within chisel3 and firrtl

New Record superclass breaks testers.

chipsalliance/chisel@2e6444c causes:

[error] .../ucb-bar/chisel-testers/src/main/scala/chisel3/iotesters/OrderedDecoupledHWIOTester.scala:394: type mismatch;
[error]  found   : chisel3.core.Record
[error]  required: chisel3.Bundle
[error]     (which expands to)  chisel3.core.Bundle
[error]     io_info = new IOAccessor(device_under_test.io)
[error]                                                ^
[error] .../ucb-bar/chisel-testers/src/main/scala/chisel3/iotesters/SteppedHWIOTester.scala:172: type mismatch;
[error]  found   : chisel3.core.Record
[error]  required: chisel3.Bundle
[error]     (which expands to)  chisel3.core.Bundle
[error]     io_info = new IOAccessor(device_under_test.io)

Poke in PeekPokeTester always happens at negedge???

I am testing a hardware that has streaming IO. From vcd waveform, it looks like that "poke" always happens at negedge. Is there a way to poke at the posedge of the clock?

How do you add commands to VCS w/ tester options?

I thought it was via testCmd = Seq(), but it's not doing anything for me???

Verilator tests fail due to GNU Make not supporting spaces if ran in a folder with spaces

Apparently Verilator calls GNU Make at some point which doesn't work with paths with spaces in the path...

@huangchiye

Poking negative numbers appears to be a problem

I haven't had time to produce a small example but this will force me to do so

Peek and Poke API for aggregates is inconsistent and hard to use.

The current tester API for aggregates needs lots of work.
Proposal:
Find a way to support Bundle and Vector Literals.

To fix or eliminate
Current poking of Vecs assigns values from an seq: IndexedSeq[BigInt] into the Vec in reverse order, i.e
for a Vec sized 10
vec(0) <= seq(9)
...
vec(9) <= seq(0)
Yet peeking from that Vec does not reverse the elements, peek and poke are asymmetric.
seq(0) <= vec(0), etc.
When poking bundles, the order seems to be correct, i.e. the first element of the seq goes into the first element of the bundle. The peek of a Bundle returns and Map of strings to BigInts, another asymmetry between peek and poke.

Users may have worked around this but it seems worth breaking the current API to fix it.
This all gets worse when Vecs and Bundles are nested.

Currently the users should write their own synthetic peek and poke routines and not use the existing API. Aggregate literals would seem to be the most elegant

do better testing of circuit reset

Consider automatically or with some sort of api, re-application of tests after asserting reset.
If there are multiple ways of applying reset, try them.
Make this the default behavior, but provide means to turn it off if necessary

testerOptions waveform misleading?

Supposedly you can specify a file location, but seems like the VCD is just dumped into the test_run_dir/YourProjectHere directory...

Refactor backends

The various backends share quite a bit of common code. It would be good to refactor this before they diverge.

Different behavior from Chisel2 for negative peeks

There's no signed conversion, so everything comes out as unsigned BigInts, I believe [tho I'm testing w/ something that extends PeekPokeTester, but I believe this is the current behavior]. For Dsp checks, this kind of matters. I can either do the signed conversion here [to be consistent with Chisel2] or within my own DspTester, but doing it here is somewhat of an API change for people. Does anyone have any opinions? If I do it at a higher level, I feel like I'm just adding patch upon patch to things... but I also need some of these changes ASAP.

@ducky64 @chick @donggyukim /whoever else might be in charge of testers...

Compile Issues

When I run sbt test, I got unresolved dependencies:
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: edu.berkeley.cs#chisel3_2.11;3.0: not found
[warn] :: edu.berkeley.cs#firrtl_2.11;0.1-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::

When I run testers as a submodule, I got this:
/scratch/dgkim/chisel-tests/testers/build.sbt:38: error: value publishTo is not a member of scala.xml.NodeBuffer
possible cause: maybe a semicolon is missing before `value publishTo'?
publishTo <<= version { v: String =>
^

Update for new Module Hierarchy

I have some PeekPokeTesters for rocket-chip LazyModuleImps. Chisel has refactored what Module points to and added a whole hierarchy. I don't really understand what all the different modules are meant to be, but evidently LazyModuleImp is not a Module (I think it is an ImplicitModule?) so I can't use PeekPokeTesters on it anymore.

I think the fix for this is simple- wherever we have a T <: Module should be replaced with T <: BaseModule or UserModule or whatever the appropriate Module type is. @chick, do you know what the appropriate Module class is?

Post-synthesis simulation

The basic issue is that I need to do synthesis post-simulation. The old way of doing this was to generate the harness from chisel, use VCS to compile it with the post-synthesis verilog and standard cells library, and use the resulting executable (usually simv) back with the chisel infrastructure.

There are two issues, one major and one minor.

Minor: There is no way to generate the harness in a known path directory, and you have to use one of the backends to get it to work even when you do not want to run vcs or verilator.

Major: Test-command is unusable.

Define higher level chisel-testers API

Currently, chisel-testers uses implementation details to interrogate the DUT graph. We need to provide a suitable API in chisel3.

See chipsalliance/chisel#231

getWidth on a Bundle type does not return the right value in a tester(PeekPokeTester)

class LinkNode extends Bundle {
  val next = UInt(64.W)
  val count = UInt(32.W)
  override def cloneType = (new LinkNode).asInstanceOf[this.type]
}

See code below for which simulation log reports:
In Bla 96
[info] [0.001] In Foo 32

class Foo (bla: Bla) extends PeekPokeTester(bla) {
  println(s"In Foo ${(new LinkNode).getWidth}")
}

class Bla extends Module {
  val io = IO (new Bundle())
  println(s"In Bla ${(new LinkNode).getWidth}")
}

class SorterAccTester extends ChiselFlatSpec {
  "FooBla" should "simulate" in {
    chisel3.iotesters.Driver(() => new Bla,"firrtl"){ c=>
      new Foo(c)
    }should be(true)
  }
}

chisel-testers master fails to build with firrtl master

chisel-testers revision d4f29ce
firrtl revision chipsalliance/firrtl@e571ef8

After running sbt publish-local on firrtl, chisel3, firrtl-interpreter in that order, trying to build chisel-testers with "sbt publish-local" results in the following error:

[info] Set current project to Chisel.iotesters (in build file:/nscratch/edwardw/chisel-testers/)
[info] Compiling 19 Scala sources to /nscratch/edwardw/chisel-testers/target/scala-2.11/classes...
[info] Main Scala API documentation to /nscratch/edwardw/chisel-testers/target/scala-2.11/api...
[info] :: delivering :: edu.berkeley.cs#chisel-iotesters_2.11;1.2-SNAPSHOT :: 1.2-SNAPSHOT :: integration :: Fri Mar 10 02:54:52 PST 2017
[info] 	delivering ivy file to /nscratch/edwardw/chisel-testers/target/scala-2.11/ivy-1.2-SNAPSHOT.xml
[info] Wrote /nscratch/edwardw/chisel-testers/target/scala-2.11/chisel-iotesters_2.11-1.2-SNAPSHOT.pom
[error] /nscratch/edwardw/chisel-testers/src/main/scala/chisel3/iotesters/ChiselMain.scala:112: class FirrtlEmitter is abstract; cannot be instantiated
[error]         (new firrtl.FirrtlEmitter).emit(firrtl.CircuitState(chirrtl, firrtl.ChirrtlForm), writer)
[error]          ^
[error] /nscratch/edwardw/chisel-testers/src/main/scala/chisel3/iotesters/ChiselMain.scala:112: class FirrtlEmitter is abstract; cannot be instantiated
[error]         (new firrtl.FirrtlEmitter).emit(firrtl.CircuitState(chirrtl, firrtl.ChirrtlForm), writer)
[error]          ^
[error] one error found

VCS Help

@jcmartin @donggyukim anyone else -- have you guys run PeekPokeTester w/ the VCS backend? I'm getting a

../vpi.h: In member function 'virtual void vpi_api_t::tick()':
../vpi.h:26: error: 'template<class T> class sim_api_t' used without template parameters

error. Have no idea what's going on. Kind of desperate, since it looks like Verilator has a horrible bug that makes testing my design next to impossible.

Unresolved dependencies on chisel, firrtl interpreter

I just cloned chisel-testers and tried to run sbt and got the following errors:

[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: edu.berkeley.cs#chisel3_2.11;3.1-SNAPSHOT: not found
[warn] 	:: edu.berkeley.cs#firrtl-interpreter_2.11;1.1-SNAPSHOT: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn] 	Note: Unresolved dependencies path:
[warn] 		edu.berkeley.cs:chisel3_2.11:3.1-SNAPSHOT (/scratch/celio/chisel-testers/build.sbt#L15-18)
[warn] 		  +- edu.berkeley.cs:chisel-iotesters_2.11:1.2-SNAPSHOT
[warn] 		edu.berkeley.cs:firrtl-interpreter_2.11:1.1-SNAPSHOT (/scratch/celio/chisel-testers/build.sbt#L15-18)
[warn] 		  +- edu.berkeley.cs:chisel-iotesters_2.11:1.2-SNAPSHOT
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: edu.berkeley.cs#chisel3_2.11;3.1-SNAPSHOT: not found
[error] unresolved dependency: edu.berkeley.cs#firrtl-interpreter_2.11;1.1-SNAPSHOT: not found

Relatedly, I'm trying to follow (https://github.com/ucb-bar/chisel-testers/wiki/Using%20the%20PeekPokeTester) but the > test-only examples.GCDSpec throws an invalid input error.

stale black_box_verilog_files.f may cause test failures

See chipsalliance/firrtl#504

Rigidity of Verilator backend re: memory black boxes

Right now, when I try to change the FirrtlExecutionOptions at the tester level to include the ReplSeqMem pass, nothing meaningful happens. Once the black box annotations are in place, it might make sense to allow custom transforms that would've previously broken simulation to be passed down... These days, I'm kind of ok using sbt test to do generation + tests in one go, so needing to enact a separate chisel Driver (edit: ok doesn't need to be in main) makes me sad...

In general, ExecutionsManagers are used in a really weird way. It seems bad if higher level stuff can't see changes made at the lower level... and It's also bad that options you set at the higher levels might not get propagated down without any warning...

Verilator concurrency issue

EDIT: Ok, apparently it's not just this change... very confused.

Problem seems to be caused by this line in VerilatorBackend (and maybe an equivalent in VCS??):

   optionsManager.chiselOptions = optionsManager.chiselOptions.copy(
       runFirrtlCompiler = false)

What does this do??? If I remove this line, my tests pass.

Original e-mail I sent to @chick:

I pulled in a bunch of changes and tried running sbt test in dsptools, but now I'm getting a really weird bug that I can't for the life of me figure out.

Basically, all of the branches, etc. I'm using are at @ https://github.com/ucb-art/Chisel3DSPDependencies

In my case, when I run sbt-test, I get 2 errors. 1 on ParameterizedAdder (which should fail b/c I removed Saturating) and 2 on my TBSpec. Jenny, who's also using this repo, gets something like 4-5 failures...

I think it has something to do with concurrency issues? But I just can't pinpoint what's going on.

If I run sbt "test-only SimpleTB.SimpleTBSpec"

It fails on two tests:

should properly read lits with gen = fixed and expect tolerance set to 1 bit (even with finite fractional bits) *** FAILED ***
[info] false was not true (SimpleTBwGenTypeOption.scala:361)

and

should properly poke/peek io delayed 1 cycle with gen = fixed and expect tolerance set to 1 bit *** FAILED ***
[info] false was not true (SimpleTBwGenTypeOption.scala:383)

This is super weird b/c it only fails on gen types (stuff I pass in as type parameters), and only when gen = some FixedPoint. Even weirder is the fact that the test after the last mentioned one doesn't fail, even though I still use gen = fixed. In that case, the only difference is that I've removed DspReal from the module. And weirder? Is that when I comment out all of the other tests except these two and rerun, things pass...

Of course, this is only an issue with Verilator... but this is driving me crazy. Any idea what's up?

chiselMain stuff is not fully integrated with scopt

previous options parsing must be integrated with the current stuff

Sequential verilator tests compile issue (again)

Hey @edwardcwang, you closed #60, but I observe the issue is there there on master branch.

In the thread, you referenced this commit by @chick that removed the call to chisel3.Driver.createTempDirectory in file Driver.scala:
ucb-bar@ed1a1d4

If the call to createTempDirectory is the workaround, it has not been restored.

The issue seen in #60 is trivial to reproduce. Make repeated calls to your tester configured for verilator backend.

class ArbiterPeekPokeTester[T <: Data](c: Arbiter[T]) extends PeekPokeTester(c) {
  (1 to 100) foreach (x => step(1))
}
  it should "elaborate" in {
    for (backend <- "verilator firrtl".split(" ")) {
      for (n <- List(2, 4, 8, 16, 32)) {
        Driver(() => new Arbiter[UInt](n = n), backendType = backend) { c =>
          new ArbiterPeekPokeTester[UInt](c)
        }
//[..]

Peek/Poking FIRRTL-generated IO

I'm trying to have FIRRTL automatically generate a scan chain when a "scan" bundle is included in my design.

Scan bundle = list of signals I want to have custom values for.

But since the added scan signals (scan clk, enable, in, out) are not part of the original Chisel code, there's no way I can test it via the usual peek, poke mechanisms... any idea if there might be some workaround?

Testers with verbose as an argument ignore them in favor of the TesterOptions

Not sure if #49 and #48 also include this problem.

If you have a class extend PeekPokeTester(c, verbose=true), verbose seems to be ignored in favor of the tester options. I think we should probably remove verbose as an argument to the testers, but this may break a lot of things.

Intermediate firrtl files created have extension .ir instead of .fir

The standard extension used for firrtl files is .fir.

Sequential Verilator tests fail and spews data in the root folder

#32 is supposedly fixed, but in the latest revision (50224d0) this error seems to have come up again, and instead of using a test_run_dir it just puts all the generated files in the root of the repo:

# ls
build.sbt  build.sh  c.py  log.txt  project  README.md  scalastyle-config.xml  scalastyle-test-config.xml  src  target 
# sbt "test-only utils.test.DebouncerTester"

[...first test runs successfully...]

verilator --cc Debouncer.v --assert -Wno-fatal -Wno-WIDTH -Wno-STMTDLY --trace -O0 --top-module Debouncer +define+TOP_TYPE=VDebouncer +define+PRINTF_COND=!Debouncer.reset +define+STOP_COND=!Debouncer.reset -CFLAGS -Wno-undefined-bool-conversion -O0 -DTOP_TYPE=VDebouncer -include VDebouncer.h -Mdir . --exe Debouncer-harness.cpp
make: Entering directory `/home/cc/chisel-template'
g++  -I.  -MMD -I/home/cc/verilator/include -I/home/cc/verilator/include/vltstd -DVL_PRINTF=printf -DVM_TRACE=1 -DVM_COVERAGE=0 -Wno-char-subscripts -Wno-parentheses-equality -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable     -Wno-undefined-bool-conversion -O0 -DTOP_TYPE=VDebouncer -include VDebouncer.h   -c -o Debouncer-harness.o Debouncer-harness.cpp
Debouncer-harness.cpp: In member function ‘void Debouncer_api_t::init_sim_data()’:
Debouncer-harness.cpp:23:61: error: ‘class VDebouncer’ has no member named ‘io_in_1’
         sim_data.inputs.push_back(new VerilatorCData(&(dut->io_in_1)));
                                                             ^
Debouncer-harness.cpp:24:61: error: ‘class VDebouncer’ has no member named ‘io_in_2’
         sim_data.inputs.push_back(new VerilatorCData(&(dut->io_in_2)));
                                                             ^
Debouncer-harness.cpp:25:61: error: ‘class VDebouncer’ has no member named ‘io_in_3’
         sim_data.inputs.push_back(new VerilatorCData(&(dut->io_in_3)));
                                                             ^
Debouncer-harness.cpp:26:61: error: ‘class VDebouncer’ has no member named ‘io_in_4’
         sim_data.inputs.push_back(new VerilatorCData(&(dut->io_in_4)));
                                                             ^
# ls
build.sbt              Debouncer.v            scalastyle-test-config.xml  VDebouncer__ALLcls.cpp  VDebouncer_classes.mk     VDebouncer__Syms.cpp         veri_api.h
build.sh               Debouncer.vcd          sim_api.h                   VDebouncer__ALLcls.d    VDebouncer.cpp            VDebouncer__Syms.h           verilated.d
c.py  log.txt                src                         VDebouncer__ALLcls.o    VDebouncer_Debouncer.cpp  VDebouncer__Trace.cpp        verilated.o
Debouncer-harness.cpp  project                target                      VDebouncer__ALLsup.cpp  VDebouncer_Debouncer.h    VDebouncer__Trace__Slow.cpp  verilated_vcd_c.d
Debouncer-harness.d    README.md              VDebouncer                  VDebouncer__ALLsup.d    VDebouncer.h              VDebouncer__ver.d            verilated_vcd_c.o
Debouncer-harness.o    scalastyle-config.xml  VDebouncer__ALL.a           VDebouncer__ALLsup.o    VDebouncer.mk             VDebouncer__verFiles.dat

Here is the relevant portion of the test code:

class DebouncerTester extends ChiselFlatSpec {
  behavior of "Debouncer"

  def get_options(backend:String): TesterOptionsManager = {
    val options = new TesterOptionsManager{
      interpreterOptions = interpreterOptions.copy(setVerbose = false)
      testerOptions = testerOptions.copy(backendName = backend)
    }
    options
  }

  val backendNames = Array[String]("verilator")
  for ( backendName <- backendNames ) {
    it should s"do a basic 6 count (with ${backendName})" in {
      Driver.execute(() => new Debouncer(2, samples=6, percent_stable=5f/6), get_options(backendName)) { c =>
        new DebouncerBasic6Count(c)
      } should be(true)
    }

    it should s"stay high with n=1 (with ${backendName})" in {
      Driver.execute(() => new Debouncer(1, samples=10), get_options(backendName)) { c =>
        new DebouncerStayHigh(c)
      } should be(true)
    }

    it should s"stay high with n=100 (with ${backendName})" in {
      Driver.execute(() => new Debouncer(100, samples=10), get_options(backendName)) { c =>
        new DebouncerStayHigh(c)
      } should be(true)
    }
  }
}

Verilator backend breaks for multiple tests

behavior of "simple module io + lits"

  it should "read lits with gen = sint properly" in {
    dsptools.Driver.execute(() => new SimpleModule(p.genShortS, p.genLongS, includeR = true, p), optionsPass) { c =>
      new LitTester(c)
    } should be (true)
  }

  it should "read lits with gen = fixed properly" in {
    dsptools.Driver.execute(() => new SimpleModule(p.genShortF, p.genLongF, includeR = true, p), optionsPass) { c =>
      new LitTester(c)
    } should be (true)
  }

When FIRRTL interpreter is used, both tests pass. When Verilator is used, the second test starts to run, but then fails with

Enabling waves..
chisel3.iotesters.TestApplicationException: test application exit - exit code 139
	at chisel3.iotesters.SimApiInterface.throwExceptionIfDead(SimApiInterface.scala:91)
	at chisel3.iotesters.SimApiInterface.chisel3$iotesters$SimApiInterface$$mwhile(SimApiInterface.scala:100)
	at chisel3.iotesters.SimApiInterface.update(SimApiInterface.scala:194)
	at chisel3.iotesters.SimApiInterface.peek(SimApiInterface.scala:281)
	at chisel3.iotesters.VerilatorBackend.peek(VerilatorBackend.scala:336)
	at dsptools.VerboseDspTester.peek(VerboseDspTester.scala:100)

If I comment out and run individual tests by themselves, things are fine... So there's something wrong with running Verilator twice? i.e. it shouldn't throw an exception when the test application "dies," because the first one actually did finish?

Full test code @ https://github.com/ucb-art/Chisel3DSPExample/blob/master/src/test/scala/UsefulExamples/SimpleTBwGenTypeOption.scala

It's hard for me to figure out what should actually be done in this case, since throwExceptionIfDead(exitValue) was put there for some reason???

@chick any ideas?

Firrtl vs. Verilator discrepancy when testing DUT with Decoupled IO's

Example DUT:

package example

import chisel3._
import chisel3.util._

class ExampleModule extends Module {
  val io = IO(new Bundle {
    val in  = Flipped(Decoupled(Bits(16.W)))
    val out = Decoupled(Bits(16.W))
  })
  
  val delay_value = 5.U
  
  val busy = Reg(init = false.B)
  val in_reg   = Reg(init=0.U(16.W))
  io.in.ready := !busy
  
  when(io.in.valid && !busy) {
    in_reg := io.in.bits
    busy := true.B
  }
  
  val wait_counter = Reg(init=0.U(16.W))
  
  when(io.in.valid && !busy) {
    wait_counter := 0.U
  }
  
  when(busy) {
    when(wait_counter === delay_value) {
      io.out.bits := in_reg
    }.otherwise {
      io.out.bits := 0.U
      wait_counter := wait_counter + 1.U
    }
  }
  
  io.out.valid := (io.out.bits === in_reg) && (wait_counter === delay_value) && busy
  
  printf("From printf -- in: ready %d   valid %d   value %d  -- out:  ready %d  valid %d  value %d\n",
         io.in.ready, io.in.valid, io.in.bits,
         io.out.ready, io.out.valid, io.out.bits)
}

Tests:

package example.test

import chisel3._
import chisel3.util._
import chisel3.iotesters._
import org.scalatest.{Matchers, FlatSpec}

import example._

class ExampleTester(dut: ExampleModule) extends AdvTester(dut){
  wire_poke( dut.io.in.valid, 1)
  wire_poke( dut.io.in.bits, 20)
  
  val inreadySignal:BigInt = peek(dut.io.in.ready)
  println(s"From peek in.ready: $inreadySignal")
  step(1)
}

class ExampleTestFirrtl  extends ChiselFlatSpec{
  "Example" should "show decoupled port controls correctly" in {
    chisel3.iotesters.Driver(() => new ExampleModule,"firrtl"){ c =>
      new ExampleTester(c)
    }should be(true)
  }
}

class ExampleTestVerilator  extends ChiselFlatSpec{
  "Example" should "show decoupled port controls correctly" in {
    chisel3.iotesters.Driver(() => new ExampleModule,"verilator"){ c =>
      new ExampleTester(c)
    }should be(true)
  }
}

when running the test with "Verilator" backend, we get:

From peek in.ready: 1
From printf -- in: ready 1 valid 1 value 20 -- out: ready 0 valid 0 value 0

However, when running the test with "Firrtl" backend, we get:

From peek in.ready: 1
From printf -- in: ready 0 valid 1 value 20 -- out: ready 1 valid 0 value 34543

The printf result of in.ready is incorrectly with "Firrtl", and there's a discrepancy between Firrtl and Verilator for the same test.

PeekPokeTester Handling Aggregates in Bundles

Previously, in Chisel 2, it was very easy to peek/poke Vecs that were contained within a Bundle. However, in Chisel 3, that doesn't seem to be the case. In seems in this version of Peek/Poke, it expects that the entities in a bundle inherit only from the Data class, not Aggregate classes. Therefore, if I had a Bundle with a Vec as a sub-entity, I would not be able to poke or peek into that Bundle properly.

This is problem because in our project: our parameterized interface uses Vecs within Bundles on our inputs and outputs. This is preventing us from upgrading our project from Chisel 2 to Chisel 3.

I noticed some mention of this issue in #77, but there doesn't seem to have been any update to that.

Also, there was a student (@dt27182) who worked on a fix for this within the chisel-tester framework. It seems all of the work is contained within this branch.

I'd be more than happy to upstream the code and put in a pull request...

VCS backend fails on blocks with no IOs (standalone)

The code below fails to compile in VCS but works with Verilator and firrtl-interpreter.

package bug

import org.scalatest.{ Matchers, FlatSpec}

import chisel3._
import chisel3.iotesters._

class VCSBug extends Module {
  val io = IO(new Bundle)
}

class Tester(c:VCSBug) extends PeekPokeTester(c) {
  step(1)
}

class VCSTest extends FlatSpec with Matchers {
  behavior of "VCSBug"
  it should "work" in {
    chisel3.iotesters.Driver( () => new VCSBug, "vcs") { c =>
      new Tester( c)
    } should be ( true)
  }
}

class VerilatorTest extends FlatSpec with Matchers {
  behavior of "VCSBug"
  it should "work" in {
    chisel3.iotesters.Driver( () => new VCSBug, "verilator") { c =>
      new Tester( c)
    } should be ( true)
  }
}

class FirrtlTest extends FlatSpec with Matchers {
  behavior of "VCSBug"
  it should "work" in {
    chisel3.iotesters.Driver( () => new VCSBug, "firrtl") { c =>
      new Tester( c)
    } should be ( true)
  }
}

The trailing comma after the reset binding in the harness is the issue:

module test;
  reg clock = 1;
  reg reset = 1;
  always #`CLOCK_PERIOD clock = ~clock;
  reg vcdon = 0;
  reg [1023:0] vcdfile = 0;
  reg [1023:0] vpdfile = 0;

  /*** DUT instantiation ***/
  VCSBug VCSBug(
    .clock(clock),
    .reset(reset),
  );

"Right way" to pass TesterOptionsManager to iotesters.Driver / use of DynamicVariable

In the default apply(), TesterOptionsManager is hardcoded in the function and passed to the execute call.

def apply[T <: Module](
      dutGen: () => T,
      backendType: String = "firrtl",
      verbose: Boolean = false,
      testerSeed: Long = System.currentTimeMillis())(
      testerGen: T => PeekPokeTester[T]): Boolean = {

    val optionsManager = new TesterOptionsManager {
      testerOptions = testerOptions.copy(backendName = backendType, isVerbose = verbose, testerSeed = testerSeed)
    }
    execute(dutGen, optionsManager)(testerGen)
  }

Now, this is the call to excecute. Observe the use of the DynamicVariable construct and call to optionsManagerVar.withValue(..)

  private val optionsManagerVar = new DynamicVariable[Option[TesterOptionsManager]](None)
  def optionsManager = optionsManagerVar.value.getOrElse(new TesterOptionsManager)
  // [..]
    def execute[T <: Module](
                         dutGenerator: () => T,
                            optionsManager: TesterOptionsManager
                          )
                          (
                            testerGen: T => PeekPokeTester[T]
                          ): Boolean = {
    optionsManagerVar.withValue(Some(optionsManager)) {
      if(optionsManager.topName.isEmpty) {
        if(optionsManager.targetDirName == ".") {
          optionsManager.setTargetDirName("test_run_dir")
        }
        val genClassName = testerGen.getClass.getName
        val testerName = genClassName.split("""\$\$""").headOption.getOrElse("") + genClassName.hashCode.abs
        optionsManager.setTargetDirName(s"${optionsManager.targetDirName}/$testerName")
      }
      val testerOptions = optionsManager.testerOptions

In my test case, I see the passed-in value of optionsManager gets squashed. Here is a code snippet.

  val optionsManager = new TesterOptionsManager
  optionsManager.setTargetDirName("not_test_run_dir")
  optionsManager.setTopName("ChiselTop")
  optionsManager.interpreterOptions = new InterpreterOptions(writeVCD=true)
  optionsManager.commonOptions = new CommonOptions()

  Driver.execute(() => new ChiselTopWrapper(p), optionsManager) { c => new ChiselTopTester(c) }

My questions is: How to use the DyanmicVariable interface? Is this intentional or an omission. I don't see this strategy/pattern used in rocket-core or chisel3.

Verilog produced from OrderedDecoupledHWIOTester has issues with DecoupleIO data values larger than 64bits during simulation

Verilator fails on lines like this in generated verilog:
$fwrite(32'h80000002,"output test event %d testing mem_out.bits = %d, should be %d\n",_T_12,device_under_test_io_mem_out_bits,_GEN_7);
when device_under_test_io_mem_out_bits width is larger than 64 bits with this message:

%Error: Foo.v:1116: Unsupported: $fwrite of dec format of > 64 bit results (use hex format instead)

This is due to %d formatting used in OrderedDecoupledHWIOTester.scala to generate fwrite messages in verilog. In my local version I changed those to %x and it worked.

Here's the diff, but I am not sure these are the only lines that need fixes.

--- a/src/main/scala/chisel3/iotesters/OrderedDecoupledHWIOTester.scala
+++ b/src/main/scala/chisel3/iotesters/OrderedDecoupledHWIOTester.scala
@@ -315,11 +315,11 @@ abstract class OrderedDecoupledHWIOTester extends HWIOTester {

   when(controlling_port.ready && controlling_port.valid) {
     ports_referenced_for_this_controlling_port.foreach { port =>

     printf(s"output test event %d testing ${name(port)} = %d, should be %d\n",

     printf(s"output test event %d testing ${name(port)} = %x, should be %x\n",
       event_counter.value, port.asInstanceOf[UInt], port_vector_events(port)(counter_for_this_decoupled.value)
     )
     when(port.asInstanceOf[UInt] != port_vector_events(port)(counter_for_this_decoupled.value)) {

       printf(s"Error: event %d ${name(port)} was %d should be %d\n",

       printf(s"Error: event %d ${name(port)} was %x should be %x\n",
         event_counter.value, port.asUInt, port_vector_events(port)(counter_for_this_decoupled.value))
       assert(false.B)
       stop()

Refactoring and clean up

Potential things to refactor:

Eliminate expect from the Backend type. Unless you expect an implementation of expect to not just be peek and compare, the implementation probably should be separate, probably in the actual tester class.
Stop using Strings to specify backend. This can (and should) be based on the type system. A potential solution is mocked up in #74.
- If this needs to be passed in as a command line argument, it should probably be (centrally) parsed in the TesterOptionsManager or something.
- Perhaps do something similar with the PrintStream? Though an implementation would need to be careful that duplicating the test options doesn't also duplicate the stream itself.
Centralize options parsing and defaults specification. Why is chisel3.iotesters.Driver.execute specifying (even overriding?) defaults for topName and targetDirName? Especially bad since now chisel iotesters can have different behavior than other testers. I understand the desire to not drop your test generated files in ., but this should be a upstream (firrtl) fix where those variables are defined.

Does verbose not work?

I have something like [again, how to setup options would be nice]:

    val options = new TesterOptionsManager {
      testerOptions = TesterOptions(
          isVerbose = true,
          backendName = "verilator",
          // TODO: Figure out what isGenVerilog really does
          isGenVerilog = true)

    }

    dsptools.Driver.execute(() => new CordicStage(p, 0 , 1), options) { c =>
      new CordicTester(c)
    } should be (true)

It's definitely using Verilator instead of firrtl-interpreter, but it's not printing out my peeks/pokes/expects. For dspPeek,dspPoke,dspExpect, it prints out dspExpect but not dspP*.

Ok, at least with DSPTester, _verbose is definitely still false...

Edit 2: _verbose comes from iotesters.Driver's (private) optionsManagerVar, which is never set.......

freechipsproject / chisel-testers Goto Github PK

chisel-testers's People

Contributors

Stargazers

Watchers

Forkers

chisel-testers's Issues

Recommend Projects

Recommend Topics

Recommend Org