I-MISALIGN_LDST-01 assumes misaligned data access will trap. Misaligned data access traps are NOT required by the ISA spec v2.2 nor the privileged spec; it is entirely reasonable for the misaligned access to be handled in hardware.
The test should be written to succeed if either the misaligned access completes correctly with no trap, or traps with the appropriate values in the CSRs.
This issue came up when I was testing a core I designed for the RISC-V soft CPU core design contest, which requires that the core pass all 55 RV32I tests. My core implements misaligned data access in hardware, but I had to add extra stuff to my core to implement the misaligned data access traps to pass the test.

Section 2.4 on processor configuration is unclear. There are two orthogonal issues. One is that a configurable implementation has to have its options specified before it can be used. The second is that the compliance test framework has to be told what features to test. The compliance test should only cover a subset of the configuration's supported features. This implementation configuration belongs in the target environment section, as it is part of selecting the target to be tested. This section 2.4 should be about the compliance test configuration, indicating how to instruct it to test a certain set of features.

Hi,

I am creating my own RISCV core based on the RISCV spec. I am running my core against the RISCV compliance test suite and I hit a problem with the I-MISALIGN_LDST-01 test under the rv32i test suite.

It seems the test has a trap handler that does nothing but to store the mbadaddress lower 2 bits and the mcause csr to the memory location 0x2020 onwards.

Also the test will store the test data in the 0x2020 onwards. From the reference output it shows:

91a1b1c1
91a1b1c1
91a1b1c1
91a1b1c1
00000001
00000004
etc

I believe the above corresponds to the test:

Test

lw x4, 0(x3)
sw x4, 0(x2)
lw x4, 1(x3)
sw x4, 4(x2)
lw x4, 2(x3)
sw x4, 8(x2)
lw x4, 3(x3)
sw x4, 12(x2)

I can understand the part about 00000001, 00000004 onwards, but I don't get it why the 1st 4 words of the reference output is 4 times 91a1b1c1.

Because a misaligned load should cause a synchronous exception, I would expect that the misaligned 'sw' instruction is not going to be committed. Hence, shouldn't the output be the original test result filler of 0xFFFFFFFF? i.e.

91a1b1c1 // first store has no misalignment, so the store is committed
ffffffff // 2nd to 4th stores causes misalignment exception, so the instruction is not committed
ffffffff
ffffffff

Does the spec says that the instruction causing the exception should be committed first before jumping to the trap handler?

Thanks,
SC

Commenting out this line to enable more verbose test output, the I-SRL-01 test in the rv32i suite fails an assertion:

Test Begin
# Test number 1
Assertion violation: file src/I-SRL-01.S, line 49: x3(0000007f) != 0xf0000000

Looking at the code for this test case:

TEST_RR_OP(srl, x3, x28, x13, 0xf0000000, 0x7ff, 0x4, x5, 12, x6)   # Testcase 3

This expands to:

li x28, 0x7ff
li x13, 0x4
srl x3, x28, x13
sw x3, 12(x5)
RVTEST_IO_ASSERT_GPR_EQ(x6, x3, 0xf0000000)

This should shift the value 0x7ff right by 0x4 bits, which should yield the value 0x7f, which is what we see x3 contains in the assertion.

I discovered this while testing my own implementation, only to find out that OVPsim uncovers the same error under the same conditions. I think the expected value should be 0x7f.

When commenting out lines in the test that fail, we see similar errors later:

Assertion violation: file src/I-SRL-01.S, line 60: x5(00000000) != 0x1000
Assertion violation: file src/I-SRL-01.S, line 61: x6(00000765) != 0x43210000
Assertion violation: file src/I-SRL-01.S, line 62: x7(00000000) != 0xfffffffe
Assertion violation: file src/I-SRL-01.S, line 74: x10(00000000) != 0x2468
Assertion violation: file src/I-SRL-01.S, line 75: x11(08000000) != 0x0
Assertion violation: file src/I-SRL-01.S, line 76: x12(00ffffed) != 0xcc000000
Assertion violation: file src/I-SRL-01.S, line 77: x13(00000001) != 0xfffffffe
Assertion violation: file src/I-SRL-01.S, line 78: x14(0000ffff) != 0xf8010000
Assertion violation: file src/I-SRL-01.S, line 92: x19(0000007f) != 0xf0000000
Assertion violation: file src/I-SRL-01.S, line 103: x21(00000000) != 0x1000
Assertion violation: file src/I-SRL-01.S, line 104: x22(00000765) != 0x43210000
Assertion violation: file src/I-SRL-01.S, line 105: x23(00000000) != 0xfffffffe
Assertion violation: file src/I-SRL-01.S, line 117: x26(00000000) != 0x2468
Assertion violation: file src/I-SRL-01.S, line 118: x27(08000000) != 0x0
Assertion violation: file src/I-SRL-01.S, line 119: x28(00ffffed) != 0xcc000000
Assertion violation: file src/I-SRL-01.S, line 120: x29(00000001) != 0xfffffffe
Assertion violation: file src/I-SRL-01.S, line 130: x30(0000ffff) != 0xf8010000

It seems the simulator is producing correct output in these cases, but the expected values in the test are wrong. This only triggers failures without the quiet macro defined, as otherwise, the register values are written to the result area, but not asserted to contain the expected values shown here.

We have a handful of linker scripts, in riscv-test-env/{p,pm,pt,v}/link.ld. These impose certain views on section naming and memory maps. These really are target platform specific details, and so should be configured on a per platform basis. Otherwise we'll be adding a new version for every platform to be tested.

The following macro from riscv-test-env/riscv_test_macros.h

#define TEST_AUIPC(inst, destreg, correctval, imm, swreg, offset, testreg) \
    TEST_CASE(testreg, destreg, correctval, swreg, offset, \
      1: \
      inst destreg, imm; \
      la swreg, 1b; \
      sub destreg, destreg, swreg; \
      )

clobbers swreg to get an position independent result. But the macro TEST_CASE assumes that swreg points to the signature and therefore the result is written to somewhere in the code section, not the signature. Consequently, the reference signature of test rv32i/I-AUIPC-01 is unchanged from the initial all 1s and always correct, no matter if AUIPC is correctly implemented or not.

I propose to add an additional argument to TEST_AUIPC for a register that can be clobbered. I will submit a pull request soon.

When a trigger doesn't exist, the test goes to the pass label. The result is that the signature is not written to after the trigger check, and the target will fail make verify. This doesn't seem like the ideal action, since it's valid for a target to have no trigger.

For context, I'm trying to add rocket chip's Default[RV32]Config as a target, and it only has one trigger.

https://github.com/riscv/riscv-compliance/blob/master/riscv-test-suite/rv32imc/src/C.LWSP.S#L86
test_data + 252 > .word 31
It is currently passing simulators tests because they have their memory cleared before starting the test (probably).

Line 39 if illegal.S sets the SSIP bit. Implementations not supporting supervisor mode can choose to not have this bit, and thus such implementation will get stuck in the self-loop that follows.

Solution: user can provide the address or macros of the memory-mapped control registers responsible for setting MSIP as part of the compliance_test.h. These can be inserted instead of line 39 and expect the same effect then on.

I noticed that the tests do not fail immediately if there is an error in $(TARGET_SIM).
So for example in Ibex the termination is done by ; and not &&.
I was just wondering if this a feature so all the other tests are also executed.
If that is the case maybe this can be improved by using && and ignore the error with - in the calling Makefile so it is more obvious in the make output.
Or maybe this is not the intended way. I am happy to create a PR but wanted to check before what the intentions here are.

Following macros in test_macros.h are used for testing compressed instructions but do not use it directly:
TEST_CLWSP, TEST_CSWSP, TEST_CADDI16SP, TEST_CADDI4SPN.

I am curious if there is any intention to reorganize the test suites. Right now, this seems a bit superficial, but in the long run this may have an effect on this repo's maintainability. For example, the rv32si and the rv32mi suites both have separate subdirectories called rv64si which appear not to be symlinked. This has so far caused at least one issue (#44). Given this,

• why are symlinks not being used?
• why do these rv64* subdirectories not exist as their own suites? I.e., why is the rv64si directory inside of rv32si, for example, not its own test suite?
• why does this layer of indirection exist in the rv32* directories? Is the intention to eventually pull the rv64* tests out into their own suites?

Note: Beyond this superficial reorganization, some discussion of privileged/unprivileged test organization has occurred in #40.

Are GPRs ( x1- x31) reset by the CPU reset?

Host: CentOS Linux 7

Prior to the most recent change:

commit 5a978cf
Author: Neel Gala [email protected]

a fresh clone would build and run with make without any configuration. It seems the most recent change regressed something (I haven't triaged it yet):

$ riscv64-unknown-elf-gcc -v
Using built-in specs.
COLLECT_GCC=riscv64-unknown-elf-gcc
COLLECT_LTO_WRAPPER=/home/tommy/install/20200211/libexec/gcc/riscv64-unknown-elf/7.2.0/lto-wrapper
Target: riscv64-unknown-elf
Configured with: /home/tommy/boom-template/riscv-tools/riscv-gnu-toolchain/build/../riscv-gcc/configure --target=riscv64-unknown-elf --prefix=/home/tommy/install/20200211 --disable-shared --disable-threads --enable-languages=c,c++ --with-system-zlib --enable-tls --with-newlib --with-sysroot=/home/tommy/install/20200211/riscv64-unknown-elf --with-native-system-header-dir=/include --disable-libmudflap --disable-libssp --disable-libquadmath --disable-libgomp --disable-nls --src=../../riscv-gcc --enable-checking=yes --disable-multilib --with-abi=lp64d --with-arch=rv64imafdc 'CFLAGS_FOR_TARGET=-Os  -mcmodel=medlow'
Thread model: single
gcc version 7.2.0 (GCC) 

$ make
for isa in rv32i rv32im rv32imc rv32Zicsr rv32Zifencei; do \
        make -j8 --max-load=4 RISCV_TARGET=riscvOVPsim RISCV_TARGET_FLAGS="" RISCV_DEVICE=$isa RISCV_ISA=$isa variant; \
                rc=$?; \
                if [ $rc -ne 0 ]; then \
                        exit $rc; \
                fi \
done
make[1]: Entering directory `/home/tommy/riscv-compliance'
make[1]: warning: -jN forced in submake: disabling jobserver mode.
make -j8 --max-load=4 \
        RISCV_TARGET=riscvOVPsim \
        RISCV_DEVICE=rv32i \
        RISCV_PREFIX=riscv64-unknown-elf- \
        run -C /home/tommy/riscv-compliance/riscv-test-suite/rv32i
make[2]: Entering directory `/home/tommy/riscv-compliance/riscv-test-suite/rv32i'
make[2]: warning: -jN forced in submake: disabling jobserver mode.
Compile /home/tommy/riscv-compliance/work/rv32i/I-ENDIANESS-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-RF_x0-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-RF_size-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-RF_width-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-MISALIGN_JMP-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-MISALIGN_LDST-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-DELAY_SLOTS-01.elf
Compile /home/tommy/riscv-compliance/work/rv32i/I-JAL-01.elf
src/I-MISALIGN_JMP-01.S: Assembler messages:
src/I-MISALIGN_JMP-01.S:282: Error: Instruction csrr requires absolute expression
src/I-MISALIGN_JMP-01.S:287: Error: Instruction csrr requires absolute expression
src/I-MISALIGN_LDST-01.S: Assembler messages:
src/I-MISALIGN_LDST-01.S:198: Error: Instruction csrr requires absolute expression
riscv64-unknown-elf-objdump: riscv64-unknown-elf-objdump: '/home/tommy/riscv-compliance/work/rv32i/I-MISALIGN_LDST-01.elf': No such file'/home/tommy/riscv-compliance/work/rv32i/I-MISALIGN_JMP-01.elf': No such file

The Clang/LLVM compiler is stricter than GCC in complying with the RISC-V assembler specification. If I used the latest Clang/LLVM compiler to compile the benchmarks, I get a failure. Extracting an individual test shows this:

riscv32-unknown-elf-clang --target=riscv32-unknown-elf -march=rv32i -c
-o I-ENDIANESS-01.o -I ../../../riscv-test-env -I
../../../riscv-test-env/p -I ../../../riscv-target/ri5cy I-ENDIANESS-01.S

../../../riscv-target/ri5cy/compliance_io.h:187:12: error: operand must
be a bare symbol name
    la t3, 0x10000000; sw a0, (0)(t3);;

This instruction sequence comes from the expansion of one of the standard header macros.

The error is that the RISC-V standard requires the second operand to the la pseudo-instruction to be a symbol, not a constant. It just happens that GCC accepts a constant - arguably a bug in GCC, certainly an unofficial extension.

It seems to me that a compliance test suite should strictly follow the standard!

(this would also seem to indicate that no one is using Clang/LLVM with the compliance tests, which would certainly seem to be a good sanity check).

The macros are target dependent and are implemented in compliance_io.h, e.g. RVTEST_IO_ASSERT_GPR_EQ. Some registers are said to be reserved and are used by the tests.

False errors get reported by macros because the checked value is a side effect of the macro implementation. As an example rv32i/src/I-ADD-01.S line 84: the code asserts x5 is 0, but the signature checks it should be 1.

The macros are pretty useful because they basically run in many environments. The asserted value should check the real value, and not a side effect of the checking macro. If a macro is using reserved registers, then it should check it and not execute it if a conflict occurs. It is likely possible to use target specific macros that can use fixed memory locations to swap registers and thus avoid solve reserved register issues.

Last comment: by not implementing the macros, it is possible to skip it. If macros are choosen, it is however not possible to skip saving the signature to memory. It would probably save memory and accelerate tests if it was possible to disable signature saving when using self-checking macros.

Update:
The problem has been fixed in I-SW-01, but (to my knowledge) persists in I-SH-01 and I-SB-01.

Here is what i think is a bug in the I-SW-01 test.

Brief:
The test writes to an address below the .data section, and program code is overwritten.

More details:

Here is the first label that the test sets up within the .data section

# Input data section.
	.data

# Output data section.
RV_COMPLIANCE_DATA_BEGIN

test_1_res:
	.fill 5, 4, -1

In the beginning of the test, that address is saved to x5

    # Addresses for test results
	la	x5, test_1_res

Shortly afterwards, it is used in a store operation

    # Clear memory
    sw      x0, -0x4(x5)

This -0x4 gambles about not storing into the .text section.
At least for me, it overwrote my halt-simulation code, so the test never ended.

Why it might not be a problem:

One can set up linker scripts to place the data section at a safe distance.
Or one can set up the macros RV_COMPLIANCE_CODE_END and RV_COMPLIANCE_DATA_BEGIN to have some safety buffer.

From the RISC-V specification, Load the data into X0 register, it can be illegal instruction. you can change the code into below format
# Addresses for test data and results
la x21, test_C_data
la x22, test_C_res
# Test
lbu x1, 0(x21)
# Store results
sw x1, 0(x22)
RVTEST_IO_ASSERT_GPR_EQ(x1, 0x00000000)
Same like I-LBU-01 test

Line 51 tries to switch off the compressed support before proceeding with the tests.
Implementations with either of the following specs:

• do not implement MISA
• implement MISA but "C" bit is readonly and hardwired to "1"

will fail to execute the test completely.

Possible solutions:

• re-arrange instructions and trap-handler such that an implementation with compressed support traps with illegal instruction, updates signature and jumps back to next test. Similar to what is done in ma_fetch.

• Have compile time options to generate a different elf for different target devices.

if all other files in the same folder use I-xxx-01.S style then the file I-IO.S should also be named I-IO-01.S

first line of I-IO.S reads:

RISC-V Compliance Test I-ADD-01

? this is wrong/misleading

https://github.com/riscv/riscv-compliance/blob/8dac5e5d3d498eaf576611ce69c2307968cf96e3/riscv-test-env/p/riscv_test.h#L159

This is the same as riscv/riscv-test-env#12

Since there is a "CHECK_XLEN" macro,
the first word(double word) of begin_signature will set to 1 if XLEN is 64.
If XLEN is 32, the first and second word of begin_signature are 0xffff_ffff.

Looking at this section of the test leaves me confused. According my understanding, we load the symbol's address of test_E2_res into x28, subtract 4 from it and compare it against the constant 0x8000403C. This test fails for me since my symbol's address test_E2_res is not anywhere near that value.

Here the excerpt from I-SB-01.S :

    RVTEST_IO_WRITE_STR("# Test part E2 - test war hazard (address register)\n");

    # Address for test results
    la      x28, test_E2_res

    # Clear memory
    sw      x0, 0(x28)

    # Test
    li      x27, 0x89ABCDEF
    sb      x27, 0(x28)
    addi    x28, x28, -4

    RVTEST_IO_ASSERT_GPR_EQ(x27, 0x89ABCDEF)
    RVTEST_IO_ASSERT_GPR_EQ(x28, 0x8000403C) # what is this assertion here for?

The following rv32i test: I-MISALIGN_JMP-01.S and rv32mi test: ma_fetch.S will not produce a signature which matches the reference signatures if the implementation supports compressed by default. I found this while running the tests on SHAKTI's e-class core.

Now this sort-of breaks the make-flow that is present. I see two options:

1. Change the Makefrag for each folder to skip these tests when the implementation supports compressed.
2. Modify the tests to check is compressed is supported, if so generate the signature and exit.

I prefer the second option. Avoids users from hacking the make-flow.

I am making a new target of my RISC-V platform, and I encounter these compiler error in the test suite:

src/I-RF_x0-01.S:57: Error: illegal operands `xori x0,x0,0xFFFFF803'
src/I-JALR-01.S:291: Error: illegal operands `jalr x0,x3,0xFFFFFFFF'

And so on. I used Zephyr SDK 0.10.1, which provides GCC 8.3.0.

I believe that GCC is unhappy with the "sign extension" done by hand, as I could remove these errors by using corresponding negative numbers.

From the TestFormatSpec:

Tests shall not depend on tool specific features. For example test shall avoid usage of internal GCC macros (e.g. __risc_xlen), specific syntax (char 'a' instead of 'a) or simulator features (e.g. tohost) etc.

The environment files, such as p/riscv_test.h and linkers all use tohost. I am curious if, in the future, this will be changed for compatibility with targets that do not use tohost and fromhost, or if there's some other workaround for this.

An example of a test currently affected by tohost functionality is scall.S.

srli.S test included in rv32ui set doesn't have it's 'reference_output' file and also there's no error message during signature comparison.

3 months back I downloaded and have been running compliance tests, things were fine.
Updated test have LOCAL_IO_PUTC(a0); in the test case causing failure.
What is the expectation here from the test case? What is testing here by this in all test cases? Please comment on this.

FN_WriteStr:
    mv          t0, a0;
10000:
    lbu         a0, (t0);
    addi        t0, t0, 1;
    beq         a0, zero, 10000f;
    **LOCAL_IO_PUTC(a0);**
    j           10000b;
10000:
    ret;

In I-MISALIGN-JMP-01, mcause is written into the signature (e.g. second value of signature). The provided signature file assumes that mcause is zero (instruction address misaligned). While this makes sense, the misaligned address also triggers another synchronous 'illegal instruction' exception in my implementation because some nonsense instruction is read from that misaligned address. According to RISC-V Privileged Architectures V20190608, Table 3.7 on page 38, the 'illegal instruction' exception has a higher priority than the 'instruction address misaligned' exception and is therefore reported, leading to a failing test.

The target environment section should explain that software is likely a component of the target environment. I would like to use the phrase "target test execution environment". The "execution environment" for a RISC-V system is something that is actually defined in Section 1.2 of the user ISA manual. It would be good to build on that here.

Line 145 and 146 in the riscv-test-env/p/riscv_test.h should be optional and not mandatory.

The mplementations not supporting Supervisor mode will not have satp register and thus would trap again. Probably the macro RVTEST_ENABLE_SUPERVISOR can be used for this.

Similarly, setting/reading PMP should be optional based on user defined inputs - not sure if there is a macro to do this.

I find that when I run the spike tests, not all of the 64-bit tests run. All that seems to run are the 64-bit tests having a "W" in the instruction name, that is, those that are not the namesake of a 32-bit test, such as those in riscv-test-suite/rv64i/src. In contrast, most of the 64-bit instructions that are the namesake of a 32-bit instruction do not seem to be tested.

I find the following directory that seems to contain 64-bit tests: riscv-test-suite/rv32ui/rv64ui . This does not seem to participate in the testing of spike. How come? I cannot find how to run these tests. How do I run all of the 64-bit tests through, say, spike?

Am I missing something obvious here? I could just find a way to hack this in, but it seems that there must be an official way to do this and I would rather use that if there is one.

Perhaps this is related to #30 , which seems to suggest that the 64-bit tests are not even finished?

Related issue: #5
https://github.com/riscv/riscv-compliance/blob/d8e99ee55d5fa3460c0a2f0898dd917a616763fb/riscv-test-suite/rv32imc/src/C-SWSP.S#L86

The test performs a write to offset 252, which is outside of section test_data (32 words).

The ma_fetch test has been written in a way that implementations with and without "C" support should generate the same signature. The aim is to avoid

j fail

at multiple places.

However, currently the signatures generated by implementation with and without C support are different. This is because the trap handler sets the return address in mepc to point to

SWSIG(?, TESTNUM)

instead of

li TESTNUM, ?

This merge request fixes that and also updates the signature. Has been verified with e-class, riscvOVPsim and spike.

I just wanted to raise awareness that the RISC-V compliance suite currently seems not respect coarser alignment restrictions on the BASE field for mtvec of a given target implementation.

The minimum alignment mandated by the RISC-V Privileged Spec v.1.11 is 4-byte but it explicitly allows for coarser restrictions:

If mtvec is writable, the set of values the register may hold can vary by implementation. The value in the BASE field must always be aligned on a 4-byte boundary, and the MODE setting may impose additional alignment constraints on the value in the BASE field.

An implementation may have different alignment constraints for different modes. In particular, MODE=Vectored may have stricter alignment constraints than MODE=Direct.

Allowing coarser alignments in Vectored mode enables vectoring to be implemented without a hardware adder circuit.

Ibex requires mtvec to be 256-byte aligned. Since the RISC-V compliance suite currently requires a finer alignment, the I-EBREAK and I-ECALL tests keep failing.

We are not sure what your view on this is, but we thought this should maybe be tracked here. For further information, please also have a look here lowRISC/ibex#428, lowRISC/ibex#99, lowRISC/ibex#100.

From line 280, the beginning of the trap handler has:

_trap_handler:
    # increment return address
    csrr    x30, mbadaddr
    addi    x30, x30, -2
    csrw    mepc, x30

The privileged spec states, "The mtval register replaces the mbadaddr register in the previous specification."

It also states, "When a hardware breakpoint is triggered, or an instruction-fetch, load, or store address-misaligned, access, or page-fault exception occurs, mtval is written with the faulting e�ffective address... For other exceptions, mtval is set to zero..."

So within the trap handler, mtval is used in place of mbadaddr, but the behavior of the two do not match up, since the spec does not actually define mtval behavior during misaligned jumps.

What I'm seeing is mtvec (i.e. mbadaddr in the test) being set to 0x0 as the spec says, so then mepc gets set to some large invalid address, which causes mret to return back into the trap handler, and then gets stuck in a loop.

Hi,
I executed breakpoint test on sweRV core (https://github.com/chipsalliance/Cores-SweRV) and it failed. After some digging and asking the community (chipsalliance/Cores-VeeR-EH1#33) I found out, that mstatus.mie must be set in order for success full execution. Therefore I simply added the following lines to the test:

# set mstatus.MIE=1 (enable M mode interrupt)
li      t0, 8
csrrs   zero, mstatus, t0

After that the test passes.

This behavior is described in Section 5.1, ‘Native M-Mode Triggers’ of the RISC-V External Debug Support specification.
Do you agree and would you add this to the test?
Best regards
cr

I think this is a similar issue to fix for ma_fetch, but in rv32si.

When I run the test with target spike with --isa=rv32ic, the signature I get is:

00000001
ffffffff
00000002
00000003
00000004
00000005
00000006
00000007
ffffffff
  .
  .
  .

Which does not match up with the signature produced with --isa=rv32i, which matches the current reference signature.

I had some problems when porting the compliance tests to my RISC-V implementation. Finally, I understand the (in my opinion) weired interaction of macros. In this post I discribe the problems I faced and what I would change to simplify the structure and increase portability.

After porting RV_COMPLIANCE_HALT, half of the tests passed and half failed. The failing tests are the old tests that were imported from riscv-tests. The newer tests, explicitly written for riscv-compliance all passed.

The signature of the failing tests all differ only in the first word: it should be 1 but it was -1. The reason for that is that in most ports RV_COMPLIANCE_HALT maps to RVTEST_PASS which in turn uses SWSIG to write 1 to the first word of the signature. My implementation of RV_COMPLIANCE_HALT did not include RVTEST_PASS nor SWSIG and therefore did not write the 1.

However, the reference signatures of the newer tests do not start with a 1, because the correct value of the first word matters. How is that implemented? The old and the new tests differ in an include file. While the old tests include aw_test_macros.h, the newer tests include riscv_test_macros.h or
test_macros.h. The former defines SWSIG to write to the signature, while the latter define SWSIG as empty.

Consequently, including SWSIG in my definition of RV_COMPLIANCE_HALT solved the problem. However, I think that is a dirty solution because it breaks the independence between test suite, test environment and target. It adds a requirement to the implementation of RV_COMPLIANCE_HALT that should at least be documented. Even better this dependency should be removed, which can be done by the following modification:

Remove SWSIG from RVTEST_PASS and move it to TEST_PASSFAIL. That works, since the new tests are terminated by RV_COMPLIANCE_HALT (without. SWSIG), while the old test are terminated by TEST_PASSFAIL (with SWSIG), which in turn calls RV_COMPLIANCE_HALT. The empty SWSIG definitions in riscv_test_macros.h and test_macros.h can be removed.

When testing this modification, I found another (related) bug: TEST_PASSFAIL writes 1 to the first word, no matter if the test passed or failed. However, the definitions of RVTEST_PASS and RVTEST_FAIL imply that in the case of an error, the failing test number should be written. Therefore I propose to copy the code from these two macros to TEST_PASSFAIL (the labels are more important than the bne):

#define TEST_PASSFAIL               \
        bne x0, TESTNUM, pass;      \
fail:                               \
        sll TESTNUM, TESTNUM, 1;    \
        or TESTNUM, TESTNUM, 1;     \
        j 1f;                       \
pass:                               \
        li TESTNUM, 1;              \
1:                                  \
        RVTEST_SYNC;                \
        SWSIG (0, TESTNUM);         \
        RV_COMPLIANCE_HALT          \

I tested the modifications in my local repo. So, when the modifications are accepted, I can provide a pull request.

If a target simulator (defined with TARGET_SIM) fails to execute, this error should be reported clearly.
A reason for an error could be that the path to the simulator is incorrect, or an argument causes an error.
Currently each test is executed without reporting the failure of execution.
The signature comparison should fail, if no previous results are present in the output folder.
Not reporting the actual error can cause some misunderstanding to the actual problem.

@allenjbaum as discussed in #70 and #80

The Atomic suite in this repository is labeled 'ua'. Are there any machine mode only Atomic extension compliance suite tests?

In running the regression on Spike, I've come across a mismatch in the Spike execution. In particular for the I-FENCE.I-01 test at line 67 (reference below), Spike executes:

(0x001101b3) instead of (0x00000137)

It is something related on how Spike implements the FETCH instruction?

https://github.com/riscv/riscv-compliance/blob/2657d67b049af1dc4bff40e82d652ed0f2ef9aee/riscv-test-suite/rv32i/src/I-FENCE.I-01.S#L67

in release 0.1, file riscv-test-suite/rv64i/Makefrag, variable rv64i_sc_tests contains "SRAIW" twice; this is still the case in commit 13e1b57

Makefile:49: target 'SRAIW.elf' given more than once in the same rule

The RV32I 'store byte' and 'store halfword' instruction tests expect that they will be able to store data at a negative offset from the test_1_res label, which is located immediately after the RV_COMPLIANCE_DATA_BEGIN label:

SB: https://github.com/riscv/riscv-compliance/blob/master/riscv-test-suite/rv32i/src/I-SB-01.S#L66

SH: https://github.com/riscv/riscv-compliance/blob/master/riscv-test-suite/rv32i/src/I-SH-01.S#L66

I believe that this can (and should) cause failures if the RV_COMPLIANCE_DATA_BEGIN section is placed at the very beginning of RAM. If RAM starts at 0x20000000, these instructions attempt to store data in address 0x1FFFFFFD, which will likely fail or trigger a trap.

If I change the linker script to start the RAM address space at 0x20000004, the tests pass as expected. Should an extra word of data be added before the test_1_res section if a negative offset will be used with that label?

Thanks.

Hi

Our team is considering to use Syntacore SCR-1. Syntacore provided a pre-built toolchain and the version is 7.1.0. However, we compiled a newer version 8.1.0. Recently, we ran the RISC-V compliance test but we got 8 test cases failed which are:

compliance_I-LB-01
compliance_I-LBU-01
compliance_I-LH-01
compliance_I-LHU-01
compliance_I-LW-01
compliance_I-SB-01
compliance_I-SH-01
compliance_I-SW-01

After tracing the cause, we located the toolchain generated some strange assembly codes for the following pseudo-instruction:

la x10, test_A4_data + 2048

8.1.0
55c:       00000f97            auipc     t6,0x0
560:       5a4f8f93             addi       t6,t6,1444 # b00 <begin_regstate>
564:       000fa183            lw           gp,0(t6)
568:       004fa203            lw           tp,4(t6)
56c:       008fa283            lw           t0,8(t6)
570:       b8418513          addi       a0,gp,-1148 # 12cc <_end+0x36c>

The correct address of x10 (a0) should be 0x12cc, however, the above codes make x10 to 0xFFFFFB84. This value causes ASSERT to the core and makes the test failed. So, I did 2 trial tests:

1. Use Syntacore pre-built toolchain (7.1.0) to compile the code
2. Modify the pseudo-instruction and compile with 8.1.0

The results are:

1. The assembly codes look prefect and all the tests are passed. As the compiler is different, the correct value of x10 (a0) is 0x12ec.

7.1.0

570: 00001517 auipc a0,0x1
574: d7c50513 addi a0,a0,-644 # 12ec <_end+0x38c>

2. Modify from "la x10, test_A4_data + 2048" to the following and rebuilt with 8.1.0:

la x10, test_A4_data
addi x10, x10, 1024
addi x10, x10, 1024

570:       00000517          auipc     a0,0x0
574:       55c50513          addi       a0,a0,1372 # acc <test_A4_data>
578:       40050513          addi       a0,a0,1024
57c:       40050513          addi       a0,a0,1024

The value of x10 (a0) is 0x12cc which is correct. And, all the test cases are passed as well.

=============================================================
a. May I ask if the newer version of RISC-V GNC toolchain support this syntax "la rd, symbol + offset"??
b. Why the old version of the toolchain can generate correct assemble codes but not in newer version? Can you help to verify this issue for me?

Thank you very much!
Jason

A few minor issues found in the initial release:

1. The top level README.md has a reference to doc/README.md which does not exist (should it be doc/README.adoc instead?)

2. The setting of the RUN_TARGET macro in the file riscv-target/riscvOVPsim/device/rv32i/Makefile.include uses the path $(ROOTDIR)/riscv-ovpsim/bin/Linux64/riscvOVPsim.exe for the executable, which does not work on a Windows host (the /Linux64/ needs be changed to Windows64 for Windows hosts).

3. If I run with RISCV_TARGET=spike or RISCV_TARGET=Codasip-simulator and the target executable is not on my path I do not get a useful error message.

4. Since there is a directory named riscv-test-suite/rv32im I tried running make RISCV_TARGET=riscvOVPsim RISCV_DEVICE=rv32im, but this does not work. This appears to be because the top level Makefile hard codes ISA=rv32i. However changing this still does not work because the directory riscv-test-suite/rv32im does not have a Makefile or other files that are in riscv-test-suite/rv32i. It seems this directory should either be completed or deleted.

Tests c-jalt and c-jr store value from link register directly into signature. Signature should not be dependant on program location in memory.

In the doc/ directory when I type "make" I get this:

Version number of title and document history do not match

This happens in both release version 0.1 and in the current master HEAD at commit 13e1b57