Refactoring RISC-V emulation APIs for easier adoption and porting about rv32emu HOT 30 OPEN

I would like to invite @RinHizakura, @qwe661234, and @visitorckw to join the discussion and contribute to the refinement of the API.

from rv32emu.

I would like to introduce cycles_per_step into state_t structure since it can be varied. For example, in web-based simulation, the user might want to increase the cycles_per_step to jump quicker to the desired part of execution to see the register file or memory bank status. For better abstraction, it could be possible to add a pair of getters and setters.

Then, rv_step signature can be refactored to have only one parameter: void rv_step(riscv_t *rv). The cycles_per_step can be retrieved via rv->userdata

I agree. By the way, state_t might not be a very self-explanatory name. I am considering unifying it into a VM-specific structure.

from rv32emu.

Aforementioned that memory I/O handlers are rarely changed, it makes less sense to define them during runtime (see main.c). Instead, I believe it is preferable to link them during build time. If really want to change the implementations, then create a new implementation and link it during build time might be a better choice. Also, from @RinHizakura previous comment, providing I/O handlers could be redundant if no much use cases want to simulate memory on their own. Thus, we might considering to move I/O handlers to library side for simplicity. Or, still providing I/O interface for registration but binding to default I/O handlers when no I/O handlers are specified on creating RISC-V instance .

For further integration of semu, we might also need to abstract the common operations among MMU and no-MMU, e.g., load and store.

The I/O improvements proposal is as below:

• riscv_io_t defined in "riscv.h" can be reused to adapt mmu_fetch, mmu_load, mmu_store in semu
  • mmu_fetch signature of semu is compatible with riscv_mem_ifetch by removing the vm and value parameter. The I/O interface is embedded inside riscv_t so vm parameter is no longer needed. The fetched value is returned
  • mmu_load signature of semu is compatible with riscv_mem_read_w, riscv_mem_read_s and riscv_mem_read_b by removing the vm, width, value and reserved parameter. The I/O interface is embedded inside riscv_t so vm param is no longer needed. The width parameter is not necessary since there are width related handlers(riscv_mem_read_w, riscv_mem_read_s and riscv_mem_read_b). The loaded value is returned. The registration of the 'reservation set' can be done in corresponding RVOP()(some fields might be added to riscv_t, e.g., reservation) so reserved parameter is no longer needed
  • mmu_store is similar to mmu_load
• Disable registration of custom I/O handlers for now and the corresponding I/O handlers are set when calling rv_create, thus the rv_create signature becomes riscv_t *rv_create(riscv_user_t attr). We can determine which I/O handlers should be bind by checking attr->data
• The peripheral I/O handlers(e.g., UART, PLIC) can be defined riscv_io_t too and they are used by RISC-V system emulator but ignored for RICS-V instructions emulator

from rv32emu.

Does hooking them at initialization still necessary in this way?

Not necessary. Let's proceed.

from rv32emu.

I am wondering shall we abstract the FILE defined in state_new as a parameter of state_new. Without abstraction, the emulator always depends on standard io(e.g., stdin, stdout, stderr). What if the user want to use a file instead of stdout?

In the initial stages of developing this emulator, I redirected I/O operations to files for comparison purposes. However, I now recognize that this approach to the function interface was not as flexible as I had initially thought.

from rv32emu.

I am wondering shall we abstract the FILE defined in state_new as a parameter of state_new. Without abstraction, the emulator always depends on standard io(e.g., stdin, stdout, stderr). What if the user want to use a file instead of stdout?

In the initial stages of developing this emulator, I redirected I/O operations to files for comparison purposes. However, I now recognize that this approach to the function interface was not as flexible as I had initially thought.

So, it could be useful to provide an abstract way for defining the desired stdin, stdout, stderr, or more than just three of them. stdin, stdout, stderr can be set as default if any spefication of them is not given.

from rv32emu.

Since memory I/O handlers are rarely changed, it makes less sense to define them during runtime (makes porting difficult). Instead, I believe it is preferable to link them during build time.

I think the distinction between modules is a little bit unclear in the current design of rv32emu. On the current design, if we regard riscv.c as the part of the library and main.c as the part of the application using the library. Although rv_create() seems to allow the application to customize memory operations through io in a pointer manner, the operation on simulated memory actually must be bound to the instance created by state_new(), which is belongs to the library side. This leads to limitations for customizing io. For example, what if you want to use a backup file to simulate memory? This design seems to make memory operations using function pointers io redundant.

I believe it is preferable to link them during build time.

So, if it doesn't matter to provide user-specific operations on memory, providing them on the build time for the library will also be a great solution.

from rv32emu.

mem_size is used for memory_new because different runtimes may have memory requirements (for example, the page size in WebAssembly is 64KiB), the default MEM_SIZE(2^32 - 1) is not appropriate for that.

Not quite sure about whether changing the memory size directly is safe. As I remember, some implementations of rv32emu intensionally rely on the fact that the memory size is 2^32 - 1 to have some trick. Or maybe I mix up with some project else. Looking for others to answer the question.

from rv32emu.

mem_size is used for memory_new because different runtimes may have memory requirements (for example, the page size in WebAssembly is 64KiB), the default MEM_SIZE(2^32 - 1) is not appropriate for that.

Not quite sure about whether changing the memory size directly is safe. As I remember, some implementations of rv32emu intensionally rely on the fact that the memory size is 2^32 - 1 to have some trick. Or maybe I mix up with some project else. Looking for others to answer the question.

The built-in ELF programs do not seem to need a lot of memory so I think 2GB - 4GB is a safe region. Dynamically changing the memory size in different runtime might be needed. For example, 64KiB multiples should be used in WebAssembly. The MEM_SIZE is set to 2^32 originally in #151 as the memory size for preallocating memory to prevent extra checking when manipulating the memory region. Then, MEM_SIZE is set to 2^32 - 1 in #221 to compatible with emcc which default build target is wasm32 ( memory shall < 4GB ).

from rv32emu.

I would like to introduce cycles_per_step into state_t structure since it can be varied. For example, in web-based simulation, the user might want to increase the cycles_per_step to jump quicker to the desired part of execution to see the register file or memory bank status. For better abstraction, it could be possible to add a pair of getters and setters.

Then, rv_step signature can be refactored to have only one parameter: void rv_step(riscv_t *rv). The cycles_per_step can be retrieved via rv->userdata

from rv32emu.

rv_enables_to_output_exit_code could be renamed as something like rv_get_xxx. Same rules might be applied to other fields of state_t to improve consistency. rv_set_xxx can be the setter.

For example:

• rv_get_userdata / rv_set_userdata
• rv_get_pc / rv_set_pc
• rv_get_reg / rv_set_reg
• rv_get_halt_status / rv_set_halt_status
• rv_get_cycle_per_step / rv_set_cycle_per_step
• rv_get_output_exit_code_flag / rv_set_output_exit_code_flag
• rv_get_allow_misalign_flag / rv_set_allow_misalign_flag
• ...

from rv32emu.

The repository mnurzia/rv serves as an additional reference for API refinement. It features three main APIs:

• Memory Access Callback: This function processes data as input/output and returns RV_BAD in case of a fault. It's defined as:
  typedef rv_res (*rv_bus_cb)(void *user, rv_u32 addr, rv_u8 *data, rv_u32 is_store, rv_u32 width);
• CPU Initialization: This function initializes the CPU and can be called again on the cpu object to reset it. The function signature is:
  void rv_init(rv *cpu, void *user, rv_bus_cb bus_cb);
• CPU Single-Step: This function advances the CPU by one step and returns RV_E * in case of an exception. Its definition is:
  rv_u32 rv_step(rv *cpu);

These APIs collectively provide a structure for memory access, CPU initialization, and step-wise execution in the CPU simulation.

from rv32emu.

As previously suggested, the maximum memory (MEM_SIZE) of a virtual machine (VM) shall be determined by the application. If these modifications are made, the Makefile-defined default stack size shall also be adjusted.

Makefile:

# Set the default stack pointer
...
CFLAGS += -D DEFAULT_STACK_ADDR=0xFFFFE000
# Set the default args starting address
CFLAGS += -D DEFAULT_ARGS_ADDR=0xFFFFF000
...

Thus, adjusting stack size should be a part in public API.

from rv32emu.

I would like to introduce cycles_per_step into state_t structure since it can be varied. For example, in web-based simulation, the user might want to increase the cycles_per_step to jump quicker to the desired part of execution to see the register file or memory bank status. For better abstraction, it could be possible to add a pair of getters and setters.
Then, rv_step signature can be refactored to have only one parameter: void rv_step(riscv_t *rv). The cycles_per_step can be retrieved via rv->userdata

I agree. By the way, state_t might not be a very self-explanatory name. I am considering unifying it into a VM-specific structure.

I think vm_attr_t can be the candidate for the name ( inspired by pthread_attr_t ).

Currently, vm_attr_t should consist of the following:

1. vm RAM size (if previous concern is OK)
2. vm STACK size (if vm RAM size changes)
3. vm-specific argc, argv
4. error code (to represent the exit state of vm)
5. enable_outout_exit_code
6. logging level
7. union of target ELF program and target vm

union {
    rv_struct_t rv_struct;
    vm_struct_t vm_struct;
};

typedef struct rv_struct {
     char *elf_program;
} rv_struct_t;

typedef struct vm_struct {
    kernel_img;
    dtb;
    rootfs_img;
} vm_struct_t ;

8. cycle_per_step
9. enable_misaligned

I would like to introduce the sixth attribute of vm_attr_t which allows the user to select how vm should log, just like printk log level of Linux kernel. This logging level will register corresponding handler during rv_init initialization. This feature enable the user has more flexibility to observe the vm state or error reporting. The sixth attribute of vm_attr_t allows to differentiate RISC-V program or RISC-V system emulation, then rv_create return a corresponding internal structure (riscv_internal or vm_internal), of course they are forward declaration structure.

Prefix of all vm-related functions should be consistent ( more discussion ).

from rv32emu.

state_t might not be a very self-explanatory name. I am considering unifying it into a VM-specific structure.

I think vm_attr_t can be the candidate for the name ( inspired by pthread_attr_t ).

It sounds promising. Please send pull request(s) to refine APIs.

Currently, vm_attr_t should consist of the following:

1. vm RAM size (if previous concern is OK)
2. vm STACK size (if vm RAM size changes)
3. vm-specific argc, argv

How about envp?

4. error code (to represent the exit state of vm)
5. enable_outout_exit_code

enable_outout_exit_code looks hacky. Can you show something detailed?

from rv32emu.

1. vm RAM size (if previous concern is OK)
2. vm STACK size (if vm RAM size changes)
3. vm-specific argc, argv

How about envp?

Since the envp is not accessible for now, place a TODO in vm_attr_t might be decent.

5. enable_outout_exit_code

enable_outout_exit_code looks hacky. Can you show something detailed?

It is related to syscall_exit to determine whether to output the exit code. I think always output the exit code is not a bad thing, maybe this is redundant. Or, it can be determined on top of logging feature.

from rv32emu.

I think always output the exit code is not a bad thing, maybe this is redundant. Or, it can be determined on top of logging feature.

After streamlining the API, we can control the exit code by storing it in a specific structure, instead of displaying it directly in the console.

from rv32emu.

I am wondering shall we abstract the FILE defined in state_new as a parameter of state_new. Without abstraction, the emulator always depends on standard io(e.g., stdin, stdout, stderr). What if the user want to use a file instead of stdout?

In the initial stages of developing this emulator, I redirected I/O operations to files for comparison purposes. However, I now recognize that this approach to the function interface was not as flexible as I had initially thought.

So, it could be useful to provide an abstract way for defining the desired stdin, stdout, stderr, or more than just three of them. Standard stdin, stdout, stderr can be set as default if any spefication of them is not given.

For abstracting file or file descriptor, we could have an attribute called bool use_default_stdin_stdout_stderr in vm_attr_t which will use common stdin, stdout and stderr. For alternative, we could have a function called vm_register_stdxxx makes the vm to register the non-common fd (e.g., regular file) before emulation.

from rv32emu.

For abstracting file or file descriptor, we could have an attribute called bool use_default_stdin_stdout_stderr in vm_attr_t which will use common stdin, stdout and stderr. For alternative, we could have a function called vm_register_stdxxx makes the vm to register the non-common fd (e.g., regular file) before emulation.

Since we should also have to maintain the file descriptor if vm_register_stdxxx() for redirection, how about just adding three file descriptors in vm_attr_t:

struct {
    ...
    int stdin;
    int stdout;
    int stderr;
} vm_attr_t;

Note: The variable's name should be refined. I just can't come up with a suitable and concise name now

These file descriptors are assigned to 0, 1, and 2 by default, and modified if vm_register_stdxxx() it. So we don't need a redundant variable bool use_default_stdin_stdout_stderr for this feature.

from rv32emu.

For abstracting file or file descriptor, we could have an attribute called bool use_default_stdin_stdout_stderr in vm_attr_t which will use common stdin, stdout and stderr. For alternative, we could have a function called vm_register_stdxxx makes the vm to register the non-common fd (e.g., regular file) before emulation.

Since we should also have to maintain the file descriptor if vm_register_stdxxx() for redirection, how about just adding three file descriptors in vm_attr_t:

struct {
    ...
    int stdin;
    int stdout;
    int stderr;
} vm_attr_t;

Note: The variable's name should be refined. I just can't come up with a suitable and concise name now

These file descriptors are assigned to 0, 1, and 2 by default, and modified if vm_register_stdxxx() it. So we don't need a redundant variable bool use_default_stdin_stdout_stderr for this feature.

Thanks for tips! I think the boolean really redundant since vm_register_stdxxx() could overwrite them.

from rv32emu.

Aforementioned that memory I/O handlers are rarely changed, it makes less sense to define them during runtime (see main.c). Instead, I believe it is preferable to link them during build time. If really want to change the implementations, then create a new implementation and link it during build time might be a better choice. Also, from @RinHizakura previous comment, providing I/O handlers could be redundant if no much use cases want to simulate memory on their own. Thus, we might considering to move I/O handlers to library side for simplicity. Or, still providing I/O interface for registration but binding to default I/O handlers when no I/O handlers are specified on creating RISC-V instance .

For further integration of semu, we might also need to abstract the common operations among MMU and no-MMU, e.g., load and store.

from rv32emu.

• mmu_fetch signature of semu is compatible with riscv_mem_ifetch by removing the vm and value parameter. The I/O interface is embedded inside riscv_t so vm parameter is no longer needed. The fetched value is returned
• mmu_load signature of semu is compatible with riscv_mem_read_w, riscv_mem_read_s and riscv_mem_read_b by removing the vm, width, value and reserved parameter. The I/O interface is embedded inside riscv_t so vm param is no longer needed. The width parameter is not necessary since there are width related handlers(riscv_mem_read_w, riscv_mem_read_s and riscv_mem_read_b). The loaded value is returned. The registration of the 'reservation set' can be done in corresponding RVOP()(some fields might be added to riscv_t, e.g., reservation) so reserved parameter is no longer needed
• mmu_store is similar to mmu_load

The proposal sounds great. I wonder how mmu_{fetch,load,store} can be interoperated with existing structure. Can you show more about function prototypes?

from rv32emu.

I wonder how mmu_{fetch,load,store} can be interoperated with existing structure. Can you show more about function prototypes?

Sure.

We have to emulate the peripherals, like MMU, UART and PLIC for minimum requirements to boot Linux.

First of all, we shall support MMU for more resource-management technique in kernel, for example memory sharing or copy-on-write(COW) such that user space programs can call fork system call. In order to support MMU, we can reuse the riscv_io_t inferface for I/O operations. The new function pointer for MMU_{fetch, load, store} might look like this:

typedef struct {
    /* memory read interface */
    riscv_mem_ifetch mem_ifetch;
    riscv_mem_read_w mem_read_w;
    riscv_mem_read_s mem_read_s;
    riscv_mem_read_b mem_read_b;

    /* memory write interface */
    riscv_mem_write_w mem_write_w;
    riscv_mem_write_s mem_write_s;
    riscv_mem_write_b mem_write_b;

    /* TODO: add peripheral I/O interfaces */

+   /* MMU memory helper interface */
+   riscv_mmu_mem_walk mmu_mem_walk;

+   /* MMU memory read interface */
+   riscv_mem_ifetch mmu_mem_ifetch;
+   riscv_mem_read_w mmu_mem_read_w;
+   riscv_mem_read_s mmu_mem_read_s;
+   riscv_mem_read_b mmu_mem_read_b;

+   /* MMU memory write interface */
+   riscv_mem_write_w mmu_mem_write_w;
+   riscv_mem_write_s mmu_mem_write_s;
+   riscv_mem_write_b mmu_mem_write_b;

    /* system */
    riscv_on_ecall on_ecall;
    riscv_on_ebreak on_ebreak;
    riscv_on_memset on_memset;
    riscv_on_memcpy on_memcpy;
} riscv_io_t;

We can decide which function pointer to call during instruction decoding stage since we will know the data width at that time.

mmu_mem_walk is the helper function to walk the 3-level page table(Sv32) with virtual memory and return the corresponding PTE. It's riscv_mmu_mem_walk interface might be like this:

typedef riscv_word_t *(*riscv_mmu_mem_walk)(riscv_word_t addr);

You might notice that non-mmu {fetch, read, write} and mmu {fetch, read, write} are duplicated after this changes. To preserve the mnemonic of the function pointers, we might want to separate them although we could use union to wrap them up to save memory. But, for simplicity, I would like not to use union first.

from rv32emu.

You might notice that non-mmu {fetch, read, write} and mmu {fetch, read, write} are duplicated after this changes. To preserve the mnemonic of the function pointers, we might want to separate them although we could use union to wrap them up to save memory. But, for simplicity, I would like not to use union first.

The proposed mmu_mem_{read,write}_[wsb] are confusing since we already have the ones prefixing with mem_. Can you avoid such inconsistency?

from rv32emu.

You might notice that non-mmu {fetch, read, write} and mmu {fetch, read, write} are duplicated after this changes. To preserve the mnemonic of the function pointers, we might want to separate them although we could use union to wrap them up to save memory. But, for simplicity, I would like not to use union first.

The proposed mmu_mem_{read,write}_[wsb] are confusing since we already have the ones prefixing with mem_. Can you avoid such inconsistency?

What about remove mem_? If so, the proposed would becomes:

typedef struct {
    ...

    /* TODO: add peripheral I/O interfaces */

+   /* MMU memory helper interface */
+   riscv_mmu_mem_walk mmu_walk;

+   /* MMU memory read interface */
+   riscv_mem_ifetch mmu_ifetch;
+   riscv_mem_read_w mmu_read_w;
+   riscv_mem_read_s mmu_read_s;
+   riscv_mem_read_b mmu_read_b;

+   /* MMU memory write interface */
+   riscv_mem_write_w mmu_write_w;
+   riscv_mem_write_s mmu_write_s;
+   riscv_mem_write_b mmu_write_b;

    ...
} riscv_io_t;

from rv32emu.

What about remove mem_?

Yes, I anticipate removing the legacy memory callback functions prefixed with mem_ in favor of the newly-added MMU counterparts. Additionally, I am considering the possibility of eliminating the mmu_{read,write} callback functions within the definition and registration of riscv_io_t. Can you refine it accordingly?

from rv32emu.

What about remove mem_?

Yes, I anticipate removing the legacy memory callback functions prefixed with mem_ in favor of the newly-added MMU counterparts. Additionally, I am considering the possibility of eliminating the mmu_{read,write} callback functions within the definition and registration of riscv_io_t. Can you refine it accordingly?

Originally, I am not intend to make mmu_{load,store} registratable, but to reduce the number of parameters that passed to a mmu_{load,store} function. However, if we want to eliminate registration for mmu_{load,store} from riscv_io_t, we can declare and define them as static functions within file scope inside "emulate.c" since all instructions implementation will be expanded by RVOP macro. In this way, the function prototype for mmu_{load,store} and helper function might look like this:
load:

static riscv_word_t mmu_ifetch(riscv_t *rv, riscv_word_t addr);
static riscv_word_t mmu_read_w(riscv_t *rv, riscv_word_t addr);
static riscv_half_t mmu_read_s(riscv_t *rv, riscv_word_t addr);
static riscv_byte_t mmu_read_b(riscv_t *rv, riscv_word_t addr);

store:

static void mmu_write_w(riscv_t *rv, riscv_word_t addr, riscv_word_t data);
static void mmu_write_s(riscv_t *rv, riscv_word_t addr, riscv_half_t data);
static void mmu_write_b(riscv_t *rv, riscv_word_t addr, riscv_byte_t data);

MMU helper function:

static riscv_word_t *mmu_walk(riscv_t *rv, riscv_word_t addr);

Obviously, we can pass a variable to indicate the width of the data and reduce the number of MMU related functions but I believe one function does one thing well might be a better adoption. Also, they might show more consistency upon the existing riscv_io_t callback functions.

from rv32emu.

Additionally, I am considering the possibility of eliminating the mmu_{read,write} callback functions within the definition and registration of riscv_io_t. Can you refine it accordingly?

One thing to notice is that: after the commit 8355777, the I/O interface are binding during initialization, thus no opportunity is given for user registration. Similar situation for mmu_{load, store} callback functions.

from rv32emu.

Obviously, we can pass a variable to indicate the width of the data and reduce the number of MMU related functions but I believe one function does one thing well might be a better adoption. Also, they might show more consistency upon the existing riscv_io_t callback functions.

Agree. Prior to the refinement of memory operations, I was thinking of Duff's device to unify these functions with various widths. However, we can benefit from compiler optimizations by using specialized functions which are not exposed and would be only hooks during initialization.

from rv32emu.

Obviously, we can pass a variable to indicate the width of the data and reduce the number of MMU related functions but I believe one function does one thing well might be a better adoption. Also, they might show more consistency upon the existing riscv_io_t callback functions.

However, we can benefit from compiler optimizations by using specialized functions which are not exposed and would be only hooks during initialization.

Yes, declare MMU related functions using static storage-class-specifier and inline function-specifier has potential to optimize them by inlining them via compiler optimization and do not expose them. Does hooking them at initialization still necessary in this way?

from rv32emu.