Performance degredation accessing large memory arrays about cocotb HOT 6 CLOSED

Have you done any profiling to find out why this could be? And are you sure this is not an issue with the simulator?

from cocotb.

I've tried it out with Verilator as the simulator and did not see the same kind of problem. I also ran with questa a wrapper file on top of the dut to emulate the assignment of the memory in the same way as was done in cocotb (but without using the VPI) and saw no performance degradation issues there. I think its a VPI issue but how to tell if its a problem on the CocoTB side of things or the questa side I'm not sure. I'm also not sure how I could just test the Questa side of the VPI.

When I ran the test again with MLEM_L2 = 16 and the cocotb profiler enabled (COCOTB_ENABLE_PROFILING=1) I get the following call graph, that might be useful.

I don't fully understand it but it looks like there's a lot of calls to the GPI handler from Scheduler._do_writes.
Looking at the code of the function:

async def _do_writes(self):
    """An internal coroutine that performs pending writes"""
    while True:
        await self._writes_pending.wait()
        if self._mode != Scheduler._MODE_NORMAL:
            await self._next_time_step

         await self._read_write

        while self._write_calls:
            handle, (func, args) = self._write_calls.popitem(last=False)
            func(*args)
        self._writes_pending.clear()

It looks to me like we make a call to each handle in the _write_calls list. I wonder if its possible to only use one handle for a logic array (or maybe that's infeasible, pointless or extremely problematic) or somehow parallelize the write calls to each GPI handler?

from cocotb.

I wonder if its possible to only use one handle for a logic array

That is already done. Unless you are explicitly doing something different.

somehow parallelize the write calls to each GPI handler?

The VPI, VHPI, FLI are probably not designed for that, and the GPI certainly isn't. Also there's no guarantee the same handle will appear in the list twice.

I wonder if the time reported in set_signal_value_int is weird due to how profiling is collected or if it's really taking that long, which would be truly absurd.

I've tried it out with Verilator as the simulator and did not see the same kind of problem. I also ran with questa a wrapper file on top of the dut to emulate the assignment of the memory in the same way as was done in cocotb (but without using the VPI) and saw no performance degradation issues there. I think its a VPI issue but how to tell if its a problem on the CocoTB side of things or the questa side I'm not sure. I'm also not sure how I could just test the Questa side of the VPI.

I feel that this means the problem is the simulator, not cocotb. I would imagine that some simulators implement large arrays as sparse arrays or minimal length vectors until they are used. I'm guessing every write (or N writes) is causing a reallocation and copy and what you are doing is just hitting a performance corner case in Questa.

Does profiling other simulators show the same weird graph where 95% of the time is spend in set_signal_value_int?

from cocotb.

I feel that this means the problem is the simulator, not cocotb. I would imagine that some simulators implement large arrays as sparse arrays or minimal length vectors until they are used. I'm guessing every write (or N writes) is causing a reallocation and copy and what you are doing is just hitting a performance corner case in Questa.

Right, I didn't really think about how the simulator might be implementing large arrays, I guess I assumed that it would be allocated at the start.

Does profiling other simulators show the same weird graph where 95% of the time is spend in set_signal_value_int?

No it doesn't seem like it. At least when I look at the profiling with verilator as the simulator the graph is very different and only 1.5% of the time is spent in set_signal_value_int:

ok, so maybe it is just a performance issue with Questa then.

from cocotb.

I'd definitely file a bug with Questa if you can. I feel like this isn't too odd of a case and it shouldn't be pessimising it.

from cocotb.

Unless there is any other questions or issues, I'll close this. Feel free to reopen if needed.

from cocotb.