Comments (28)
I agree that MiSoC should become a real python package. I also agree it's sometimes difficult to group things in topics. When it's ambiguous I don't mind trying to flatten things. But for example:
- LitePCIe is clearly a communication core. You will maybe access a memory through it but then it's a use case for this core and it does not preclude that LitePCIe is still a communication core. Same for LiteUSB, LiteEth.
- LiteSATA is also clearly a memory core. If lower layers were not specific, I totally agree they should live in communication. But here everything is specific to this application:
- You are not going to reuse the transceiver code since it has more than 300 parameters that have to be configured specifically for SATA application.
- Control of the transceiver is also very specific to SATA applications (OOB, speed negociation). Upper layers are also specific the SATA specification (Link, Transport, Command layers)
So there is no interest in reusing anything from LiteSATA for high speed serial communications. It can be interesting in the future to develop a generic low level core to create a high speed serial communication link between 2 FPGA, but even in that case you are not going to reuse that for LiteSATA.
So I'd like to keep "big" cores as self contained and organized in topics (maybe outside of MiSoC it it prevents packaging MiSoC as you want to do) For small cores that do not belong to existing topics, we can try to flatten things (soc, cpu, tools).
About the cores organization:
- phys are the hardware equivalent to software backends.
- core implements the main fonctionnality or the core.
- frontend provides modules on top of the core that are used by the user (DMAs, bridges, crossbar, test modules such memtest or a bist, ...).
About moving software, examples_designs, doc outside of cores, from this:
liteeth
- core
- doc
- examples_designs
- frontend
- phy
- software
- test
litepcie
- core
- doc
- examples_designs
- frontend
- phy
- software
- test
liteusb
- core
- doc
- examples_designs
- frontend
- phy
- software
- test
I have the impression we will endup with something like this:
doc
- liteeth
- litepcie
- liteusb
software
- liteeth
- litepcie
- liteusb
example_designs
- liteeth
- litepcie
- liteusb
tests
- liteeth
- litepcie
- liteusb
Except maybe for packaging MiSoC, I don't see real advantages for that compared to the actual organization and I'm not sure I want to split the big cores like that.
So we have to find the best compromise for Migen and MiSoC in the globality.
As we already discussed we should move all bus, csr generation, complex modules from Migen to MiSoC. And maybe move "big" cores outside of MiSoC (as I initially started to do) if having them self containted is not compatible with what you want to do.
We can then have:
In Migen: All that is needed to generate HDL and build a design.
In MiSoC: All that is needed to generate a SoC and the basic cores/tools for that.
And the "big" cores outside of that.
Or finally maybe something that would be in fact easier for everyone:
Only one global repository (and give it a name) with subdirectories:
- migen
- mibuild
- miplatforms (For me mibuild shoud not provides platforms directly).
- misoc
- micores
And package each of this indidually to allow that:
from misoc import soc
from miplaforms import kc705
from micores imports litepcie
What do you think of that?
(My answer is in fact more a reflexion that has evolved since I was writing...)
Florent
from misoc.
Fine with me. If it would be in one big repo I fail to see how exporting five top-level packages is helpful (nobody else in the python ecosystem does it that way AFAICT) but that is secondary. IMHO splitting into repositories only becomes useful if there is a release of one, if the dependencies become inconvenient, or if the repositories become huge. So far we tend to demand that all of mi* is in sync anyway.
I grant you the labeling of pcie as communication and sata as memory. But is that the dominant and obvious category? What about serial busses versus parallel? Where does an 8b10b codec go? Do I understand you correctly that you are OK with dropping the topics? Otherwise I would have asked you to weigh the advantages of having allegedly related components side-by-side against the confusion of misplacing cores and having related cores far apart. I would claim that somebody looking for a sata core will look -- well -- for a sata core and not be inclined start browsing memory cores (spiflash, norflash those are not orthogonaly as well). There would still be a grouping of the cores by tags (or topics) in the documentation/README. Then you can relate cores side by side by different metrics much better.
The same reasoning for/against works in the case of frontend/phy/core/... directories. It is really hard to slot stuff in there and much harder to predict where you will find what you are looking for and what you will find if you peek into a directory. The need for a directory should be completely obvious. Splitting something into frontend/core/phy is usually not reliable and future proof as in the case of the UART where we now have a frontend/core that tries to adapt itself to idiosyncrasies originating in the phys (synchronous versus asynchronous, non-full versus almost-empty). It is probably only reliable if the interface between the two is defined/standardized.
If we get the documentation (there is very little, and stub documentation files that bounce a search for information with a request to contribute the very same are not helpful) into one common tree, generating common output from it becomes much cleaner. If the examples are not inside the misoc tree, it becomes much harder to litter the tree with build/
directories.
from misoc.
I'm OK to flatten things if we can keep the "big" cores in a separate directory (cores directory in misoc, or micores out of misoc).
We would have something like that without topics:
(mi)cores:
- litescope
- liteeth
- litepcie
- litesata.
- ...
And if each "big" core keep the organization it already has with its doc, example designs and so on.
But each core can have its own organization if it's more relevent. Things that are not specific to the core and that can be reused by others cores are placed into misoc. (8b10b, generic packetizers, etc...) New cores can be designed in misoc and moved in cores when they are big enough.
Also what about moving targets from misoc and then have something like that? :
- (mi)gen: Elements to generate HDL.
- (mi)build: Elements to build a design and handle vendor's tools.
- (mi)soc: Element to build a SoC. (flattened actual misoc - "big" cores + csr, bus,... from actual migen)
- (mi)cores: "big" cores.
- (mi)platforms: Supported platforms
- (mi)targets: Targets for the supported platforms.
Or eventually group platforms and targets in a same directory.
from misoc.
Sure. Once a component needs multiple files, it gets a directory. But it is an obligation to be concise and export its stuff in an organized fashion using e.g. __init__.py
.
Allowing "the big cores" to cram everything into their own directory is probably not a good idea. If these cores refuse to integrate with the rest of the misoc package (documentation, executables, examples, tests, packaging) they can just as well be maintained outside misoc. Wasn't the idea to try to keep misoc from becoming a dump where everybody is king of his own directory-empire like opencores?
Moving cores around as a standard operating procedure is promising to break everybodies code.
Why is there a need to draw a line between soc
and cores
?
There is a good reason to keep the platforms with migen: Otherwise you don't get a useful package out of migen and you will always have to look around for platform definitions. With the platforms and mibuild in migen, you can start right away and don't have to juggle three things.
from misoc.
So it seems we both came to the conclusion that the "big" cores must live outside misoc.
Merging them in misoc was maybe not a good idea since they can in fact be seen as others project using misoc as a tool to build them or to integrate them in others design.
I'm going to create a repository with all the litexxx cores (and probably call it litecores) and then remove them from misoc. I will maybe just keep the ethernet mac if you want to avoid external dependency.
Then we'll be able to reorganize the code. Is it fine for you?
from misoc.
I would still love to see the big cores in misoc, just integrated nicely in terms of documentation, examples, tests, executables, and packaging.
But for the reorg, sure. This is a pretty big thing. How do we schedule/track it?
from misoc.
I'm sorry but no: I don't want to split these cores. If that's not compatible with the new organization, lets move them outside of misoc.
For the reorganization, we should maybe create a fork for that and then merge it in master once stabilized.
from misoc.
Hmm. That is unfortunate. What could be done is -- as you suggested -- have an area for cores that are not integrated, maybe as git submodules, and then merge their code with misoc using the namsepace-package mechanism. Then you can still from misoc import litescope
even if litescope
is not contained/shipped in misoc.
from misoc.
Yes that would be fine.
from misoc.
For me, the documentation, example designs, tests and hdl all together constitute the core. That's why I want to keep them together in a single directory. I know this is not what you have in mind, but let's try to find a compromise since this is the only thing I'd like to keep untouched, all your others ideas are fine for me.
I've been thinking about a simpler reorganization:
Keep migen and misoc separated but move bus and bank from migen to misoc.
Have a global organization like this:
- build
- cores
- structure
- doc
- test
- targets
Actual misoclib would be splitted in structure and cores. Things related to the misoc structure: bus, bank, soc generation, tools, make.py, cpuif.py... would go to structure. Cores would have a flattened organization without topics, something like this:
- cpu
- dvisampler
- flash
- framebuffer
- gpio.py
- spi.py
- liteeth
- liteusb
- litescope
- litepcie
- litesata
- sdram
Since actual software is in fact closely related to the cpu cores, this would be moved to cores/cpu/. We can also remove ext_cores directory and move submodules directly to the correct place: lm32/mor1kx submodules would be moved to cores/cpu/. Software would also be packaged with misoc to allow building it out of misoc directory.
The doc directory would contain the documentation for the structure and the basic cores with the tools to build it. The big cores would use the same tools for the documentation and building big cores documentation would be done from this directory. Documentation for the big cores would be generated as separated documents or integrated as chapters in a bigger documentation that will cover misoc in its globality (structure + basic cores + big cores).
The test directory would contain the tests for the structure, basic cores and we would also contain a single script that would be used to discover and run all the tests contained in cores.
The build directory would probably not be necessary if make.py is packaged with misoc.
Does it seem better for you? I'm really trying to find the best compromise for both of us :)
from misoc.
Sounds good. Ack to everything, just a few things:
- Tests can remain close to the code they test (e.g.
misoc.litescope.test
). There is no need for a global/test/
directory. Tools likenose
can find them. You will probably like that. - I would still merge your structure and cores (
misoc.wishbone
andmisoc.litesata
). And I would also be ok with havingmisoc.lm32
,misoc.mor1kx
,misoc.spiflash
, andmisoc.norflash
at the top level, unless they start sharing code. Thesdram
stuff seems so intermingled that it probably benefits from a directory. /build
should probably become./build-$target
(xilinx does not like concurrentl builds in the same directory) and withinmisoc
only be used for the targets fromtargets
, out-of-tree projects will end up containing their ownbuild-$target
- Having the verilog code scattered seems ok to me.
- Having software scattered around is problematic and error-prone because it makes search and include paths and makefile combinations really complicated.
- If you scatter the documentation, you can not build it with the usual tools. Remember, this should be able to become a real python package.
from misoc.
- If tests remain close to the code they test what do we do for the simple cores that reside in a single Python file? Integrate the test with that file?
- Where do we put all the other cores that we move from Migen, e.g.
migen.genlib.sort
,migen.genlib.mhamgen
,migen.flow.*
? - I guess what Florent meant is simply that software should be in
/cores/cpu
. But it should be fine to put it all in a/misoc/cores/cpu/software
similar to the current/software
.
from misoc.
The idea with the structure and cores directories is to split things between modules/tools that are used to build a core or a SoC and the effectives cores. Structure would contain most of the generic modules we would move from Migen. ie something like this:
misoc.structure.bus.wishbone
, misoc.structure.bank.description
, misoc.structure.soc
and for the cores:
misoc.cores.litescope
, misoc.cores.spliflash
, misoc.cores.norflash16
, misoc.cores.cpu.lm32
I'm not considering the modules used for plumbing, arbitrating, adapting,... as cores, so having the wishbone arbiter or the dataflow converters for example in structure seems fine to me.
I'm OK to flatten things in structure and cores but I have the feeling that whithout at least these 2 main topics, this will be a mess.
For the software, I don't want it to be scattered. As Sebastien said I would just move it to /misoc/cores/cpu/software
.
Others cores can also have software (ex the linux software of litepcie, or a possible future fx2 or fx3 firmware for LiteUSB) A SoC does not necessary have an embedded cpu, so its software should be treated as it's done for the others cores.
For the test of simple cores, yes we could integrate the test in that file. (I would also like to move to misoc.structure
the tools used in the testbenchs of the big cores like: packetstreamer, packetlogger, data checkers...). That would reduce the size of the integrated tests of simple cores.
I would also like to have higher level tests using verilator (or future Python simulation). For example be able to run the simple target with stubs on UART, Ethernet or others interfaces for checking. This would probably fit in misoc.test
If that's really not possible to keep the doc scattered, then I would be OK to do compromises on that.
from misoc.
- I would not hesitate to put tests for that in
misoc.test.spiflash
misoc.sort
,misoc.mhamgen
etc. I suspect problems and little benefit from delineatingstructure
,cores
,genlib
,flow
from each other reliably and in an obvious future proof way.- Where does target support code for sata, pcie, flash etc go? Also
misoc/cpu/software
and notmisoc/litesata/software
? If litesata code gets separated frommisoc/litesata
, the entire thing could just as well go to/software
.
from misoc.
Robert, are you ok if we keep the tests for small cores into the same file (which would be misoc.spiflash
in that case)? This way there will be no global test directory.
from misoc.
And I guess what Florent meant is that software for the embedded CPU (BIOS, baselib, etc.) goes to misoc/cpu/software
, whereas software associated with the other cores (e.g. PCIe driver for Linux) goes with each core. Is that right, Florent?
from misoc.
Re tests with cores in the same file: sure. that's nice.
About scattering the code: Sure. I would also put the software (python host-side code) in the core's directory. And it should be standard practice to hook up with the packaging infrastructure and generate executable scripts with the correct interpreter for CLI tools.
But about the c-code (bios, runtime, libbase): imagine you then have a project that uses a cpu, litesata, spiflash and litescope. It builds its software (the runtime that runs on the soc) out of tree (outside the misoc tree in its own working directory) as it should. Would'nt it be a bit of a mess to glob up all the makefile snippets for the different parts if the c-code is scattered?
from misoc.
Would there be a significant amount of code that can be shared between a LiteSATA driver for PC and one for the embedded CPU? And same with litescope...
from misoc.
All the build infrastructure (Makefile stuff) would be shared. And litescope will probably want to use/include/link code from misoc/cpu/software
.
from misoc.
The out-of-tree software build system? Yes, that one could go in its own directory. Do you have examples of Makefiles that do fully-out-of-tree builds to get a precise idea of how this would work?
Does litescope need anything else than generated/csr.h?
from misoc.
I had thought libopencm3 (as mentioned above) does it. They support a slew of different processors and combinations of flags. buildroot certainly has it. And it does not look that hard. .o
and generated headers would just land where the project is.
A few cores would want to use crc*
, hook up their own commands into the bios shell, use console.h
.
from misoc.
Yes Sebastien I meant that software for the embedded CPU should go in misoc/cpu/software misoc/cpu/software
. That's also why I would create a cpu directory in misoc with lm32 and mor1kx in it: If we add another cpu, we would be able to share the code in c
Not sure we understood correctly each others, but I don't want for example to move the spi_flash C code that is in software
in misoc/spiflash
, it would stay in misoc/cpu/software
, same for uart and others drivers.
Software of the others cores does not share anything with what would be in misoc/cpu/software
.
It's mostly host software scripts or drivers that only require the csr.h or csr.csv that will be generated during the build.
Now I agree that in misoc/cpu/software
, you will have code that is related to cores that are outside of that. (spi_flash, uart, dram...) but having them in the cores would probably be a mess. Potentially, you will also want have code to initialize the dram in LitePCIe or the access the flash, and it probably won't share anything with the embedded software.
So this not perfect, but seems acceptable, no?
from misoc.
Hi,
I've moved the cores that would have been difficult to refactor in a flattened organization outside of misoc. So I no longer have restrictions on what Robert was suggesting. I'm willing to help for this reorganization, but I'll wait until we will all be ready to start. (I don't want to break ARTIQ or others designs and we can probably work in a branch first as Robert suggested).
Florent
from misoc.
Any comments on the organization currently implemented in the new branch?
from misoc.
Lots of good things about the new structure!
Some things I thought after having a quick glance;
- It was nice that previously all the "external cores" where collected together into one directory. The new structure makes it hard to tell when it is external.
- The misoc/software directory feels a little bit "out of place" because it is really "data" rather than Python code.
- With there being one misoc/cores directory, the naming of cores becomes a lot more important. It feels a bit more confusing then it was previously. For example, why is sdram split into sdram_phy, sdram_{phy,mode,settings,tester}?
- Is it expected that cores in external repos would also end up placing things under the "misoc/cores" structure?
- It would be nice if we could improve the documentation on the cores :)
from misoc.
- What do you mean by "external" cores and why does it matter? lm32, or1k, etc. are git submodules and installed/distributed with misoc.
- this is a requirement for making it part of an installable python package.
- the sdram structure is: sdram_settings (like dfi) is shared by controllers and PHYs, and then the other cores are largely independent. How would you organize this?
- external repos never modify misoc, which becomes a regular python package. If you need a core which is not in misoc, you should import it from its own repository (the "targets" are now executable files, which makes things more flexible here).
from misoc.
The few bits in migen that I have played bit so far look good. What would help is a porting.rst
or NEWS.rst
that describes what has changes and where the new things are. Also (and that would be a point for NEWS.rst
) I have not jet figured out where Source
, Sink
and all the actor stuff have gone.
from misoc.
It's in misoc.interconnect.stream
(and simplified).
from misoc.
Related Issues (20)
- Wrong version in `setup.py`
- different clocking of differentiator and integrator HOT 3
- misoc@2022-04-07 fail with either latest migen built from source (2022-09-02) and with migen 0.9.2 HOT 1
- pip install of tag 0.12 fails HOT 5
- fix liteeth error counter atomicity problem HOT 2
- liteeth: remove dw parameterization HOT 2
- liteeth: rename dv to rx_dv
- liteeth: remove MII clock output
- clean up memory regions, names, offsets
- kasli/spi_flash: investigate and implement quad i/o read
- DDR3-related error in Vivado timing report for Ultrascale HOT 1
- spi: first half-cycle (setup period) of a chained, pure read uses the write divider instead of the read divider
- Record.connect return value not used HOT 3
- Binutils compilation
- Standard 1 bit SPI mode for spi_flash not available? HOT 2
- software not rebuilt after gateware change
- fix or remove Ultrascale SDRAM init HOT 4
- Wanted to Build a small SoC HOT 1
- CSRStorage with write_from_dev=True seems broken in simulation HOT 3
- register_rom weirdness HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from misoc.