osmhpi / metalfs Goto Github PK
View Code? Open in Web Editor NEWNear-storage compute aware file system and FPGA operator pipelines.
Home Page: https://metalfs.github.io
License: MIT License
Near-storage compute aware file system and FPGA operator pipelines.
Home Page: https://metalfs.github.io
License: MIT License
The I/O orchestration does not receive an interrupt from the hls_streamsink
once the incoming stream is terminated. Instead, it terminates whenever the data source or any other operator have signaled completion.
When profiling, this leads to the performance counters being stopped too early, thus the profiling results do not show the entire amount of data that was transferred.
Currently card is hardcoded to card0. Card number can be changed in src/metal_pipeline/pipeline.c, line 24:
https://github.com/osmhpi/metal_fs/blob/master/src/metal_pipeline/pipeline.c#L24
You have to re-run make in the build directory afterwards.
Thanks Robert.
AC.
Processing throughput could possibly be improved by issuing read and write requests concurrently while executing the pipeline (instead of issuing separate mount/writeback jobs).
The conceptual problem here is how we can control simultaneous read/write requests to the storage controller. Maybe we can introduce a wrapping component that dispatches read and write requests issued by the two dataflow processes.
For building FPGA images for cloud platforms (e.g. Nimbix), the SNAP action must be synthesized to fit into a reconfigurable partition of the outer image controlled by the cloud provider.
Since the SNAP Action of Metal FS is a Vivado Block Design, this is currently not possible. Block designs cannot be the top component of a partial reconfiguration image.
Currently the blowfish implementation doesn't correctly handle partially-populated stream elements.
One possibility to deal with this would be a CTS mode of operation, which has the benefit that the output data has the same size as the input.
The best image I was able to build had a timing miss of around -300ps.
We need to investigate why the timing was missed and implement workarounds.
Some operators (e.g. metal_cat
and the blowfish operators) expect file paths as part of their command line arguments.
Because these paths can be relative, they have to be resolved in the handle_args
callback of the respective operator implementations, taking the working directory of the agent process into account.
Ideally, we wouldn't have to modify SNAP's action_wrapper
to:
This can be achieved by setting up a custom build process that is invoked in SNAP's action preprocess build phase. We would then rely on SNAP's hdl action infrastructure.
Points to consider:
This issue was introduced when we started writing data in 4KB bursts.
Apparently, there's not enough memory allocated on the host side (i.e. not a multiple of 4KB), so the memory operator overwrites heap metadata, which leads to an exception when calling free()
on the buffer later on. We have to make sure that our input/output buffers are always 4KB-aligned and multiples of 4KB in size.
When creating a file on the FPGA, e.g. echo "Test" | ./operators/passthrough > ./files/test
it (sometimes?) cannot be read from. When calling rm ./files/test
, FUSE responds with a "Function not implemented" error code.
I suspect this to be a FUSE-related issue because our metadata seems to be correct: When re-mounting the FS, everything seems normal. Also, when enabling FUSE's debug switch (-d
), some weird re-naming operations seem to happen.
When specifying different keys to encrypt and decrypt data using the blowfish operator, we still obtain the correct plaintext.
This could be because either
Currently there are still a few manual steps required to implement an operator:
mtl_operator_specification
by writing C code -- this could possibly be replaced by a declarative operator.json config file, e.g.:{
"name": "blowfish_encrypt",
"id": "5:3",
"help": "Usage: ./blowfish_encrypt [...]",
"options": {
"key": { "short": "k", "type": "file", "length": 128 }
}
}
Define job types for configuring the operator, this could be replaced by a generic configuration job that automatically passes the configuration data to the respective operator (possibly macro-driven)
Register the operator in both hardware (action_run_operators
) and software (known_operators
), we could also improve on this
Currently, the only integer operator parameter type is int
(which internally maps to a uint32
).
This naming should be clarified, and parameter types for uint64
, int64
, uint32
, int32
should be added.
Furthermore, the term 'parameter' is used synonymously with 'option' and 'argument' in the codebase and documentation. This should be harmonized as well.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.