chipsalliance / dromajo Goto Github PK

1. there is an exit backdoor during handling ecall. The bare metal test case may not follow the sbi specification, I coincidentally triggered it, and I think it may be an unnecessary feature leftover from the development process.
   

   dromajo/include/dromajo_template.h

   Lines 1173 to 1175 in fae7b3a

   	/* Intercept SBI_SHUTDOWN, that is, ecall with a7 == SBI_SHUTDOWN */
   	if (!s->ignore_sbi_shutdown && s->priv == PRV_M && read_reg(0x17) == SBI_SHUTDOWN)
   	s->terminate_simulation = 1;

2. dromajo uses the old specification that fill 0 to tval when illegal instruction, I suggest writing the illegal value in it.
   

   dromajo/include/dromajo_template.h

   Lines 1636 to 1638 in fae7b3a

   	illegal_insn:
   	s->pending_exception = CAUSE_ILLEGAL_INSTRUCTION;
   	s->pending_tval = 0;

3. In hardware, some CSR are not guaranteed to be 0 during reset, I suggest using random values to fill them, like mie. We can see some firmware has the reset mie behavior.
   https://github.com/riscv-software-src/opensbi/blob/bd355213bfbb209c047e8cc0df56936f6705477f/firmware/payloads/test_head.S#L52

On a linux machine, I am able to complete the following steps:
Get a trivial buildroot (~ 23 min)
Get the Linux kernel up and running (~ 3 min)

However the following step and those after it are not working:
openSBI (~ ? min)

I think
1) the "opensbi.dromajo.tar.gz" tarball is unavailable
and/or
2) the following path is missing from the opensbi-0.5 repository.
opensbi-0.5/build/platform/dromajo/firmware/fw_payload.bin

TinyEMU had some support for networking/mounting devices to share data.

Is this possible to do with Dromajo? If so any guides/advice on how to do it?

This is forked off @ss2783's pull request #15 where extending the CLINT to handle sub-word accesses ran into trouble.

The fundamental problem is that the device model is currently called upon the load/stores directly and it's up to the handler to deal sub-word handling.

I think there's a model that works better:

1. Allocate backing store (RAM) to the address space taken by the device
2. All loads and stores just read the backing store as normal memory
3. There's an optional pre-read hook which is called with hints about the access address and size.
   Devices without read side-effects don't need to provide this hook.
4. There's an optional post-write hook which is called with hints about the access address and size.
   Devices without write side-effects don't need to provide this hook.
5. There's an optional timer callback that can be registered to be called for every N ticks. Alternatively, it can schedule itself on a as-needed basis, TBD.

This significantly reduces the amount of code that needs to be written to support most devices and removes the sub-word access handling from devices (unless they really want to handle this in a special way).

Hi,

While I was using dromajo, I noticed that mie has a MIE_MCIP field.

https://github.com/chipsalliance/dromajo/blob/master/src/riscv_cpu.cpp#L1215

I could not see this field in SPEC (it is reserved). Can you please let me know if this field is intentionally left there for any reason?

Thanks

I think it will be a good idea to include some regression test to this project to prevent performance degradations.
Are there any benchmarks people commonly use for testing the code?

0x80000000 represents -2147483648 when treated as 32-bit signed integer (w), but when converting to single precision floating point number, the dromajo generates the wrong result.

It can be tested with the following code:

    li t0, 0x80000000
    fcvt.s.w f0, t0
    fmv.x.w t1, f0

The expected result is 0xCF000000 (-2147483648.0) in t1, but dromajo generates 0x80000000 (-0.0).

Create a simple travis unit test (run some code) to make sure that the code is not broken for pull requests.

When more open cores use dromajo, add some simple cosim to the fast regression.

The specification (Volume II: RISC-V Privileged Architectures V20211203 Page 37) said: mepc is an MXLEN-bit read/write register formatted as shown in Figure 3.21. The low bit of mepc (mepc[0]) is always zero. On implementations that support only IALIGN=32, the two low bits (mepc[1:0]) are always zero.

csr write function has this mask operation:

Background

I am working to emulate/simulate RISC-V systems (using multiple tools) with more than 8 cores and booting SMP Linux with the OpenSBI bootloader. I currently have a setup that works with both Dromajo and QEMU (v6.1.0) to boot OpenSBI v0.9 with Linux Kernel v5.9 and a simple Buildroot/Busybox filesystem. This system successfully boots in Dromajo with up to 8 cores, but fails at higher core counts due to the hard-coded core count limit of 8 in Dromajo (

I attempted to modify the source code and increase this limit, say to 32, but running with anything more than 8 cores results in either an error (e.g, with 9 cores: riscv_cpu_read_memory: invalid physical address 0x0000000000011000) or a segmentation fault of Dromajo itself. One potential conclusion is that the Dromajo generated device tree is invalid for core counts > 8 and it isn't allocating a large enough memory range for some device.

Questions

What is the motivation for limiting Dromajo to simulating 8 cores maximum and what are the constraints or requirements for increasing the maximum core count?

Are there any short-term workarounds for running with > 8 cores (e.g., if this is indeed a device tree issue and it is possible to provide a custom device tree)?

Branch/Commit

master at 3675ba0

Misc

Note that unmodified QEMU v6.1.0 has a similar hard-coded core limit for the riscv64-virt machine, but increasing this limit and rebuilding QEMU seems to cause no problems.

Currently, Dromajo does not support checkpoint generation when run in a multicore settings.

The comments in the codebase mention of this issue. Filing it here for the documentation purposes.

The specification (Volume II: RISC-V Privileged Architectures V20211203 Page 85) said: If Svnapot is not implemented, bit 63 remains reserved and must be zeroed by software for forward compatibility, or else a page fault exception is raised.
Dromajo trace:

0 3 0x000000008000028c (0x0105051b) x10 0x0000000040201010
0 3 0x0000000080000290 (0x02a00593) x11 0x000000000000002a
0 3 0x0000000080000294 (0x00b52023)
0 3 0x0000000080000298 (0x000205b7) x11 0x0000000000020000
0 3 0x000000008000029c (0x3005b073)

Rocket/spike trace:

core   0: 0x000000008000028c (0x0105051b) addiw   a0, a0, 16
core   0: 0x0000000080000290 (0x02a00593) li      a1, 42
core   0: 0x0000000080000294 (0x00b52023) sw      a1, 0(a0)
core   0: exception trap_store_page_fault, epc 0x0000000080000294
core   0:           tval 0x0000000040201010
core   0: 0x0000000080000004 (0x34202f73) csrr    t5, mcause
core   0: 0x0000000080000008 (0x00800f93) li      t6, 8

Dromajo ignores the PTE N bit and calculates the address as a normal PTE.

For this and all other numeric options, if given as a string, we allow hexadecimal constants.

Building a bootrom and/or DTB should be a option (disabled by default?)

Could be related to this issue

When executing the following test 1406, PC= 0x8000_02dc, instruction = sc.d s11, sp, (sp) Spike (and Rocket, and BlackParrot) both raise a store access fault, but Dromajo doesn't seem to be raising this exception and continues to execute subsequent instructions from which point onwards, cosimulation fails (as an exception handler jump doesn't happen). Attached are the test binary, elf, spike simulation output, dromajo output. The DUT in the bottom right in figure is BlackParrot. (But also verified compliant behavior in Rocket). Dromajo is the only output that's off out of the other 4.

Drive folder

A screenshot to make life easier (Caption in each of the 4 command lines):

Seems to be doing this by design:

https://github.com/chipsalliance/dromajo/blob/master/src/dromajo_cosim.cpp#L251

This line executes the first instruction of the exception handler, thus it no longer shows up later when exception handling starts in the DUT.

I was luckly/ unlucky enough to actually reproduce this while running Linux:

hartid=0 priv: 0 exception user_ecall, epc 0x0000003fa17a4938
hartid=0            tval 0x0000000000000000
0 0x0000003fa17a4938 (0x00000073)                        ecall
[DEBUG] DUT raised exception 8
[DEBUG] DUT raised interrupt 7
[DEBUG] DUT also raised exception 8
[DEBUG] Interrupt: MIP <- 7: Now MIP = 80
get_irq_mask: mip=0x80 mie=0x2aa mideleg=0x222
raise_interrupt: irq=7 priv=1 pc=ffffffff80002ed0 hartid=0
hartid=0: exception interrupt #7, epc 0xffffffff80002ed0
3 0x0000000080000004 (0x34011173) x2  0x000000008000bec0 csrrw   sp, mscratch, sp
3 0x0000000080000008 (0x1a010863)                        beqz    sp, pc + 432
3 0x000000008000000c (0x04a13823)                        sd      a0, 80(sp)
3 0x0000000080000010 (0x04b13c23)                        sd      a1, 88(sp)
3 0x0000000080000014 (0x342025f3) x11 0x8000000000000007 csrr    a1, mcause
3 0x0000000080000018 (0x0805d263)                        bgez    a1, pc + 132
3 0x000000008000001c (0x00159593) x11 0x000000000000000e slli    a1, a1, 1
3 0x0000000080000020 (0x00e00513) x10 0x000000000000000e li      a0, 14
3 0x0000000080000024 (0x02b51263)                        bne     a0, a1, pc + 36
3 0x0000000080000028 (0x08000513) x10 0x0000000000000080 li      a0, 128
3 0x000000008000002c (0x30453073)                        csrc    mie, a0
3 0x0000000080000030 (0x02000513) x10 0x0000000000000020 li      a0, 32
3 0x0000000080000034 (0x34452073)                        csrs    mip, a0
3 0x0000000080000038 (0x05013503) x10 0x0000002ad48fe000 ld      a0, 80(sp)
3 0x000000008000003c (0x05813583) x11 0x0000003fa180a930 ld      a1, 88(sp)
3 0x0000000080000040 (0x34011173) x2  0x0000003fef66ae50 csrrw   sp, mscratch, sp
3 0x0000000080000044 (0x30200073)                        mret
1 0xffffffff80002ed0 (0x00021f63)                        bnez    tp, pc + 30
[error] EMU PC ffffffff80002ed0, DUT PC ffffffff80002ecc
[error] EMU INSN 00021f63, DUT INSN 14021273
[error] EMU MSTATUS 8000000a000060a0, DUT MSTATUS 00000000
[error] DUT pending exception -1 pending interrupt -1

The DUT tried to start execute the 1st instruction of the exception handler after mret, which is at 80002ecc. However the dromajo has already at the 2nd instruction into the handler, at 80002ed0. This matches the observation in the code where the 1st instruction of the exception handler is already executed.

Another fall-out of Pull Request #15: the handling of interrupt embeds too much policy into Dromajo. For co-sim, we need to be able to adapt to the DUTs decision on when to take pending interrupts. The way current way is hacked and the better way would be to simply factor out of the instruction execution all handling of interrupts. Instead we leave it to the client to example pending interrupts and raise them as needed.

In the setup.md, boot.cfg is mentioned multiple times. But I have no idea what that is and it seems to be undocumented.

Hi,

When I was try to build dromajo with simpoint flag as following, the build fails. can you please help with this issue? I'm using the latest commit: 3e95064
(HEAD -> master, origin/master, origin/HEAD)
Author: Tommy Thorn [email protected]
Date: Thu Mar 2 12:02:11 2023 -0800

dromajo/build_simpoint - [master] $ cmake -DSIMPOINT=On ../

-- The C compiler identification is GNU 8.5.0
-- The CXX compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: dromajo/build_simpoint

dromajo/build_simpoint - [master] $ make

[ 85%] Building CXX object CMakeFiles/dromajo.dir/src/dromajo.cpp.o
[ 90%] Linking CXX executable dromajo
[ 90%] Built target dromajo
[ 95%] Building CXX object CMakeFiles/dromajo_cosim_test.dir/src/dromajo_cosim_test.cpp.o
[100%] Linking CXX executable dromajo_cosim_test
libdromajo_cosim.a(riscv_cpu.cpp.o): In function `csr_write(RISCVCPUState*, unsigned int, unsigned int, unsigned long)':

dromajo/src/riscv_cpu.cpp:1878: undefined reference to simpoint_roi' dromajo/src/riscv_cpu.cpp:1880: undefined reference to simpoint_roi'
dromajo/src/riscv_cpu.cpp:1882: undefined reference to simpoint_roi' dromajo/src/riscv_cpu.cpp:1883: undefined reference to simpoint_roi'
dromajo/src/riscv_cpu.cpp:1887: undefined reference to `simpoint_roi'

Thanks.

Dromajo does not respond to maxinsns variable when ncpus > 1

To reproduce:

./dromajo --trace 0 --maxinsns 100000 --ncpus 2 [riscv-bin]

As mentioned in the #35 , I think there are some ways to format the codes from different contributors.

• There are some out-of-the-box plugins to format the c code on the actions marketplace.
• It would also be a good idea to include the standard guidelines for formatting in the README.md and add the .clang-format file to the repo.

I was trying the Dromajo setup on my Mac. I got the riscv baremetal stuff to work but the linux build is not working.

In particular I am trying this part:
Linux with buildroot
Get a trivial buildroot (~ 23 min)

I get an error. See screenshot. I installed gcc on my Mac using "brew install gcc" but it didn't help.

@abejgonzalez, this change

commit d868962
Author: abejgonzalez [email protected]
Date: Mon Nov 18 16:53:43 2019 -0800

Update for Cosimulation (Parameter Overrides + Misc. Cleanup

broke the Linux booting example in run.

Prior it booted the OpenSBI and kernel as expected, after the change we get

riscv_cpu_write_memory: invalid physical address 0x0000000010000004
sbi_trap_error: hart0: page/access fault handler failed (error -2)
sbi_trap_error: hart0: mcause=0x0000000000000007 mtval=0x0000000010000004
sbi_trap_error: hart0: mepc=0x0000000080006f18 mstatus=0x8000000a00007800
sbi_trap_error: hart0: ra=0x00000000800031d4 sp=0x0000000080014d60
sbi_trap_error: hart0: gp=0x0000000000000000 tp=0x0000000080014e00
sbi_trap_error: hart0: s0=0x0000000080014d70 s1=0x0000000000000000
sbi_trap_error: hart0: a0=0x0000000010000004 a1=0x0000000000000035
...

The method of booting the system from multiple boot ROM files, as implemented in PR #76, is not the most elegant solution.

For instance, when the system only has a single boot memory, which is usually the case, it becomes challenging to place multiple generated files into that single memory space without resorting to messy workarounds.

The proposed solution is to create a single boot code file that will be divided into N sections, where N is the number of cores. Each core will generate its own recovery code and write it to its designated section. In addition to the recovery code, each core will initiate with a preamble code that reads the hart_id and calculates the PC based on that ID.

I started implementing this, please let me know if you have any suggestions or concerns.

I haven't yet looked into why, but keyboard input takes seconds to arrive which seems far slower than the rest of the execution speed. This looks like a bug.

By default the V extension is enabled in dromajo (and reported in the misa register). There doesn't seem to be any runtime option to disable it. In the code I noticed a switch in riscv_cpu.h:

/* Uncomment the next line to DISABLE Vector Simulation "V-extension" */
//#define VLEN 0

However that doesn't help because a few lines later it would be re-enabled because now VLEN is less than ELEN:

#if (VLEN_MAX < VLEN || VLEN < ELEN || !IS_PO2(VLEN))
#undef VLEN
#define VLEN VLEN_DEFAULT
#endif

Defining ELEN to 0 as well also doesn't help as that would cause ELEN to be less than ELEN_MIN:

#if (ELEN < ELEN_MIN || VLEN < ELEN || !IS_PO2(ELEN))
#undef ELEN
#define ELEN ELEN_DEFAULT
#endif

If I patch these all out I am getting some compilation errors:

error: ‘RISCVCPUState’ {aka ‘struct RISCVCPUState’} has no member named ‘most_recently_written_vregs’; did you mean ‘most_recently_written_reg’?
  175 |             if (cpu->most_recently_written_vregs[i]) {
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                      most_recently_written_reg

I ended up just patching the code inside riscv_cpu.cpp to ask it not report MCPUID_V in misa to avoid divergence from DUT. But I am feeling like this should be something supported without modifying the code.

I ran the attached assembly program with dromajo and it gives an exception on a nop instruction. I ran the same program on Spike and it ignored nop instruction and proceeded as expected.

I have attached a zip file that contains the compiled binary file (test.elf), assembly file (test.S), and disassembly of the binary (test.txt).

The following figure shows the diff between Spike trace and dromajo trace. nop instruction is located after the highlighted mret instruction.

test.zip

It's clear that adding more command-line options aren't the path forward. What I suggest instead is that everything (or almost everything?) should be possible to specify as a parameter in a config file. All current command line options will internally translate to a hidden config file. Furthermore, one should be able to load multiple config files, to enable refactoring. Finally, it most be possible to give an inline config files as command line option. Example:

dromajo base.cfg -c '{ "resetvector" = "0x1000" }'

The config files are loaded in-order, such that later ones can override earlier ones. Finally, inline configs are loaded last.

dromajo lr/sc does not fully respect the riscv manual ?
====
The SC must fail if a store to the reservation set from another hart can be observed to
occur between the LR and SC.

write printf to include/dromajo_template.h
====
#define OP_A(size)                                                                      \
..            case 2: /* lr.w */                                                          \
                printf("lr.w: addr=%x\n", addr);                                        \
..            case 3: /* sc.w */                                                          \
..              if (s->load_res == addr) {                                              \
..                  printf("sc.w success memval=%x hartid=%x\n",                        \
                                    read_reg(rs2), s->mhartid);                         \
                    val         = 0;                                                    \
                    s->load_res = ~0;                                                   \
                } else {                                                                \
                    printf("sc.w failed hartid=%x\n", s->mhartid);                      \
                    val = 1;                                                            \
                }                                                                       \

crt.s from dromajo/run vs laur.s
====
$ diff crt.S laur.S 
127,129c127,129
<   # for now, assume only 1 core
<   li a1, 1
< 1:bgeu a0, a1, 1b
---
>   # laur: run on all cores
> #  li a1, 1
> #1:bgeu a0, a1, 1b
139c139,148
<   j _init
---
> #  j _init
> 
>   # laur
>   # have each core add 1 to tohost
>   la a1, tohost
> 1: lr.w a4, (a1)
> addi a4, a4, 1
> sc.w a4, a4, (a1)
> bnez a4, 1b
> label1:j label1

compile and run
====
laur@laurPC-100:~/lucru/cn/riscv/64/dromajo/run$ riscv64-unknown-elf-gcc -march=rv64g -mabi=lp64 -static -mcmodel=medany -nostdlib -nostartfiles laur.S -lgcc -T test.ld
laur@laurPC-100:~/lucru/cn/riscv/64/dromajo/run$ ../build/dromajo --ncpus=4 ./a.out
lr.w: addr=80001000
lr.w: addr=80001000
lr.w: addr=80001000
lr.w: addr=80001000
sc.w success memval=1 hartid=0
sc.w success memval=1 hartid=1
sc.w success memval=1 hartid=2
sc.w success memval=1 hartid=3
Simulation speed:  0.29 MIPS (single-core)
Power off.