hushenwei2000 / rvv-atg Goto Github PK

View Code? Open in Web Editor NEW

12.0 2.0 6.0 11.81 MB

RISC-V Vector Automatic Tests Generator

Python 99.95% Shell 0.05%

rvv-atg's Introduction

RISC-V Vector Automatic Tests Generator

Branches Descriptions

This repository has two available branches:

master: generates tests that contain self-checking.
new_test_format: generates tests that are used for the RISCOF framework (no self-checking, use signature for verification).

The test content is the same in both of them.

Prerequisite

RVV Compiler
1. Set gcc,objdump, and riscof directory's path variables in file: scripts/constants.py
VLEN config
1. scripts/create_test_floating/create_test_common.py _vlen = change it to your vlen
Spike
1. Install Latest https://github.com/riscv-software-src/riscv-isa-sim.
2. Add <spike_path>/build to your PATH
If the terminal can find spike command then it's successful
RISCV-ISAC RVV Support
1. git clone https://github.com/hushenwei2000/riscv-isac-rvv
2. cd riscv-isac-rvv
3. git checkout vetcor !!!IMPORTANT!!!
4. pip install . --prefix=~/.local # Anywhere you want
5. Add ~/.local to your PATH
If the terminal can find riscv_isac command then it's successful
Sail Reference Model (If you want to run RISCOF to test)
1. Intall sail-riscv with vector support https://github.com/riscv/sail-riscv. master branch
  1. sail version should be 0.16
  2. Note that you should modify vlen and elen in riscv_sys_control.sail after init_pmp() before building. Add these two lines for example:
```
vlen = 0b0011;
elen = 0b1;
```
  vlen and elen encoding can be found in riscv_vlen.sail.

Usage

Generate One Instruction

python run.py -i <instruction> -t <type> [--vlen VLEN] [--vsew VSEW]

The type shall be consistent with the instruction: i (integer), f (floating point), m (mask), p (permute), x (fix point), l (load store)
Supported instruction and type can be seen in cgfs/<type>/<instruction>.yaml
vlen VLEN Vector Register Length: 32, 64, 128(default), 256, 512, 1024
vsew VSEW Selected Element Width: 8, 16, 32(default), 64

Generate All Tests

python generate_all.py

This will use default parameter configuration to generate all integer instructions tests.
Modify runcommand_<type> function to run different parameter.
Modyfy main function to run different type of instructions.

After genering, run python move_generate_all_elf.py, this will check if generated file is valid. If you see

No first.S for vXX is OK, ignore it
No RVMODEL_DATA_END for vXX then it means vXX you should use generate one instruction method to regenerate it. Regenerate all of them and rerun move_generate_all_elf.py until no No RVMODEL_DATA_END appear.
All tests will in generate_all folder.

Run Tests On RISCOF

Modify riscof_files/env/test_macros_vector.h, you can select your prefered configuration and copy from env/macros/vsewXX_lmulXX and paste to riscof_files/env/test_macros_vector.h
cd riscof_files
Modify your reference model and DUT to be testes in config.ini, in current file, we use sail as reference and spike as DUT
Modify GCC in sail_cSim/riscof_sail_cSim.py and spike/riscof_spike.py
copy the .S test files to test_suite
Modify VLEN and ELEN argument in riscof_files/spike/riscof_spike.py line 166: -varch=vlen:512,elen:64
run riscof run --config=config.ini --suite=test_suite --env=env

Known Bugs

Not support floating-point vsew=16
Lack of vmerge tests
Lack of load store eew=64 tests
SEGMENT load and store signature problem: if eew > sew, only the first register will be signatured
mask instructions lack of test different registers tests

Support Configuration

Parameter	Numbers	Current Support	Note
FLEN	FP16, BF16, 32, 64	32, 64
vlen	128 ~ 2^16	128 ~ 1024	Spike now support largest 4096
vsew	8, 16, 32, 64	All
lmul	1/8, 1/4, 1/2, 1, 2, 4, 8	All	lmul=8 except load/store， 1/8 except eew>=32 loadstore
vta	0, 1	0
vma	0, 1	0

Notice of Failure Tests

elen = 64 for default
- vsew should <= elen * lmul
- So, when lmul = 0.125, load/store eew=32+ cannot test.
If there are failure tests, most probably is because that configuration can not be tested on that instruction. For example test widen instruction when vsew=64, or test floating point when vsew=8/16, or the elen related restriction above.

Signature

For arithmetic instructions that has vv, vx, vi types, each type tests store signature into x12, x20, x24, respectively.

rvv-atg's People

Contributors

Stargazers

Watchers

Forkers

nadiga-ventana hamnamohi quswarabid syedhassanulhaq oxyw heiyuen1999

rvv-atg's Issues

Invalid test generation for vaadd instruction with vlen = 512, LMUL = 8, SEW = 8 configuration

I am generating a test for vaadd fixed point instruction with the following configuration:
vlen = 512, LMUL = 8, SEW = 8

I used following command to generate the test:

python3 run.py -t x -i vaadd --vlen 512 --lmul 8 --vsew 8 --xlen 64 --flen 64

My understanding is that if it is generating a test for a specific instruction (in this case vaadd), there would be at least one execution of vaadd but I don't see any execution of this instruction for this specific configuration.
While for the remaining configurations it is working as expected i.e. it has executed the instruction which is being tested:

core   0: 0x0000000080000260 (0x25042c57) vaadd.vv v24, v16, v8, v0.t
core   0: 0 0x0000000080000260 (0x25042c57) c8_vstart 0x0000000000000000 e8 m1 l64 v24 0x00811ecf1c452e0ae82d552b8b445536307d97442d004d0374da352ed0d72d04b49ae55d9ed4e107c4d9b11b112e3f0367f9ab10fd2ce801f68219ef66da64fe

Is there any known bugs related to this?

Note: I have checked only vaadd instruction, may be there are more cases like this.

riscv_isac coverage: Error: No such option: -v

I am learning how to use this project and try to generate testcases with command python run.py -i vadd -t i and it failed with error below

$ python run.py -i vadd -t i

Tue, 07 Nov 2023 10:27:08 INFO     Creating output directory: 11-07-vadd-vlen128-vsew32-lmul1.0-masked
Tue, 07 Nov 2023 10:27:08 INFO     Finding CGF for: vadd in /home/hachi/sail/rvv-atg/cgfs/i/vadd.yaml.
Tue, 07 Nov 2023 10:27:08 INFO     lmul: 1.0.
vta, vma  False False
new_vtavma:  tu, mu
Tue, 07 Nov 2023 10:27:08 INFO     RVV-ATG: instr: vadd, vlen: 128, vsew: 32, lmul: 1.000000
Tue, 07 Nov 2023 10:27:08 INFO     Creating empty test for vadd
cp: '11-07-vadd-vlen128-vsew32-lmul1.0-masked/vadd_empty.S' and '11-07-vadd-vlen128-vsew32-lmul1.0-masked/vadd_empty.S' are the same file
Tue, 07 Nov 2023 10:27:08 INFO     Creating empty test for vadd: finish in 11-07-vadd-vlen128-vsew32-lmul1.0-masked/vadd_empty.S!
Tue, 07 Nov 2023 10:27:08 INFO     Running riscof coverage: vadd.empty
Tue, 07 Nov 2023 10:27:08 INFO     Running riscof coverage: vadd.empty, stage: Compiling...
riscv64-linux-gnu-gcc -march=rv64gv    -w     -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles                 -T /home/hachi/sail/rvv-atg/riscof_files/spike/env/link.ld                 -I /home/hachi/sail/rvv-atg/env/macros/vsew32_lmul1                 -I /home/hachi/sail/rvv-atg/env/p                 -I /home/hachi/sail/rvv-atg/env                 -I /home/hachi/sail/rvv-atg/env/spike                 -I /home/hachi/sail/rvv-atg/env/sail_cSim -mabi=lp64          vadd_empty.S -o ref_empty.elf -DTEST_CASE_1=True -DXLEN=64 -DFLEN=32;
Tue, 07 Nov 2023 10:27:08 INFO     Running riscof coverage: vadd.empty, stage: ObjDumping...
Tue, 07 Nov 2023 10:27:08 INFO     Running riscof coverage: vadd.empty, stage: Spike Running...
riscv_isac --verbose info coverage -d                         -t spike_vadd_empty.log --parser-name spike -o coverage_empty.rpt                         --sig-label begin_signature  end_signature                         --test-label rvtest_code_begin cleanup_epilogs                         -e ref_empty.elf -c /home/hachi/sail/rvv-atg/cgfs/dataset.yaml -c /home/hachi/sail/rvv-atg/cgfs/i/vadd.yaml -x64 -v128 --vsew 32 --lmul 1 --flen 32 -l vadd > isac_log_empty 2>&1;
Tue, 07 Nov 2023 10:27:08 INFO     Running riscof coverage: vadd.empty, stage: RISCV-ISAC Running...
Tue, 07 Nov 2023 10:27:08 INFO     Running riscof coverage vadd finish!
Tue, 07 Nov 2023 10:27:08 INFO     Creating first test for vadd
Traceback (most recent call last):
  File "/home/hachi/sail/rvv-atg/run.py", line 183, in <module>
    main()
  File "/home/hachi/sail/rvv-atg/run.py", line 170, in main
    run_integer(cwd, args, cgf,  output_dir)
  File "/home/hachi/sail/rvv-atg/run.py", line 106, in run_integer
    first_test = create_first_test(
                 ^^^^^^^^^^^^^^^^^^
  File "/home/hachi/sail/rvv-atg/scripts/lib.py", line 118, in create_first_test
    return eval(func_str)
           ^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
  File "/home/hachi/sail/rvv-atg/scripts/create_test_integer/create_test_vadd.py", line 42, in create_first_test_vadd
    rs1_val, rs2_val = extract_operands(f, rpt_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hachi/sail/rvv-atg/scripts/test_common_info.py", line 193, in extract_operands
    f = open(rpt_path)
        ^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/hachi/sail/rvv-atg/11-07-vadd-vlen128-vsew32-lmul1.0-masked/coverage_empty.rpt'

It seems that the riscv_isac command failed with error below

$ cat /home/hachi/sail/rvv-atg/11-07-vadd-vlen128-vsew32-lmul1.0-masked/isac_log_empty

    INFO | ****** RISC-V ISA Coverage 0.18.0 *******
    INFO | Copyright (c) 2020, InCore Semiconductors Pvt. Ltd.
    INFO | All Rights Reserved.
Usage: riscv_isac coverage [OPTIONS]
Try 'riscv_isac coverage --help' for help.

Error: No such option: -v

The riscv-isac is the latest version from https://github.com/riscv-software-src/riscv-isac . Thanks in advance.

Modifications required in generate_all.py script

The generate_all.py script is there to generate tests for all instructions. However the script is not thorough enough to create all tests. Two points needs to be addressed regarding this:

All the instructions should be covered along with all its legal combinations. Therefore, a regressive test generation script is desired.
All tests covering all instructions with all its variants and configuration should be placed in a directory which have a structure consistent with riscv-tests. This includes all the test placed in one folder (rv64uv in this case) with a makfrag listing all those tests that can be included in test environment (riscv-test-env) later. Header files (whether scalar or vector) should be included in a single directory (rv64uv/macros) and no changes should have to be made in the environment itself (or at the worst, least changes are desired) that uses those tests after compiling according to any target, spike or DUT etc.

I'll try to submit a PR very soon to address this problem. I'd prefer if this generation script doesn't alter the current generate_all.py script, and can be used as a standalone script to address just this problem. I'd appreciate your comments on it.

Regards,
Quswar Abid

ERROR in generation of all tests (using generate_all.py code)

After setting up all that is required, I am getting following error:

Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
Traceback (most recent call last):
  File "run.py", line 3, in <module>
  File "run.py", line 3, in <module>
        from ast import arg
from ast import arg
ImportErrorImportError: : cannot import name argcannot import name arg

Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
Traceback (most recent call last):
  File "run.py", line 3, in <module>
    from ast import arg
ImportError: cannot import name arg
grep: 11-04-%s-vlen512-vsew32-lmul1.0/spike_%s_final.log: No such file or directory
cp: cannot stat '11-04-%s-vlen512-vsew32-lmul1.0/ref_final.elf': No such file or directory

A quick google lookup states something that points out a problem in calling out wrong imports (in multiple places) as in

The Python "ImportError: cannot import name" occurs when we have circular imports (importing members between the same files). To solve the error, move the objects to a third file and import them from a central location in other files, or nest one of the imports in a function.

Generation of valid tests for unsupported VLEN target

It is documented in the README that the generator supports VLEN up to 1024. However it is observed that the test generator can generate tests up to VLEN of 8192. Running the following command gives generated tests.

python3 run.py -t i -i vadd --vlen 8192 --vsew 64 --lmul 1

Even python3 run.py -t i -i vadd --vlen 4096 --vsew 64 --lmul 1 and python3 run.py -t i -i vadd --vlen 2048 --vsew 64 --lmul 1 works as well. I even ran the tests with VLEN == 4096 on an RTL as well. It did not create tests for VLEN == 16384 though.

THE questions I want to ask is .. does the README need to be updated? or is there any problem that I haven't noticed?

Spec Support ?

I can't find this mentioned anywhere in the README file. So, I wanted to know that what spec does this generates tests for? is it Riscv Vector Extension 1.0?

Ease of use for setting paths of gcc, objdump, and riscof

Opening this issue along with a PR (#4) to provide ease of use to set paths from one single location. This will make it easier for users to set up this test generator.

hushenwei2000 / rvv-atg Goto Github PK

rvv-atg's Introduction

RISC-V Vector Automatic Tests Generator

Branches Descriptions

Prerequisite

Usage

Generate One Instruction

Generate All Tests

Run Tests On RISCOF

Known Bugs

Support Configuration

Notice of Failure Tests

Signature

rvv-atg's People

Contributors

Stargazers

Watchers

Forkers

rvv-atg's Issues

Recommend Projects

Recommend Topics

Recommend Org