datenlord / datenlord Goto Github PK
View Code? Open in Web Editor NEWDatenLord, Computing Defined Storage, an application-orientated, cloud-native distributed storage system
License: MIT License
DatenLord, Computing Defined Storage, an application-orientated, cloud-native distributed storage system
License: MIT License
to enable unsafe_code lint, we have to remove unsafe code, or only allow minimum unsafe code for performance consideration.
there are several trivial casts in the code, they have to be fixed before enabling trivial_casts lint.
error: trivial cast: `&T` as `*const T`
--> async_fuse/src/fuse_reply.rs:72:37
|
72 | ... let p = &instance as *const T as *const u8;
| ^^^^^^^^^^^^^^^^^^^^^
|
note: the lint level is defined here
--> async_fuse/src/main.rs:12:5
|
12 | trivial_casts, //TODO: remove trivial casts in code
| ^^^^^^^^^^^^^
= help: cast can be replaced by coercion; this might require a temporary variable
error: trivial cast: `&protocol::FuseOutHeader` as `*const protocol::FuseOutHeader`
--> async_fuse/src/fuse_reply.rs:94:21
|
94 | let h = &header as *const FuseOutHeader as *const u8;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: cast can be replaced by coercion; this might require a temporary variable
error: trivial cast: `&std::ffi::OsStr` as `*const std::ffi::OsStr`
--> async_fuse/src/mount.rs:255:42
|
255 | let res = unsafe { libc::unmount(mntpnt as *const _ as *const u8 as *const i8, MNT_FORCE) };
| ^^^^^^^^^^^^^^^^^^
|
= help: cast can be replaced by coercion; this might require a temporary variable
error: trivial cast: `&mut u32` as `*mut u32`
--> async_fuse/src/mount.rs:281:52
|
281 | let result = unsafe { fuse_read_random(fd, &mut drandom as *mut _)? };
| ^^^^^^^^^^^^^^^^^^^^^^
|
= help: cast can be replaced by coercion; this might require a temporary variable
error: trivial cast: `&mut mount::param::FuseMountArgs` as `*mut mount::param::FuseMountArgs`
--> async_fuse/src/mount.rs:346:17
|
346 | &mut mnt_args as *mut _ as *mut c_void,
| ^^^^^^^^^^^^^^^^^^^^^^^
|
= help: cast can be replaced by coercion; this might require a temporary variable
error: aborting due to 5 previous errors
Fix following clippy warning
warning: method is never used: `fetch_slice`
--> async_fuse/src/fuse_request.rs:52:5
|
52 | pub fn fetch_slice<T>(&mut self) -> anyhow::Result<Vec<&'a T>> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (8/7)
--> fuse_ll/src/fuse/mod.rs:306:5
|
306 | / fn release(
307 | | &mut self,
308 | | _req: &Request<'_>,
309 | | _ino: u64,
... |
314 | | reply: ReplyEmpty,
315 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: method is never used: `uid`
--> async_fuse/src/fuse_request.rs:698:5
|
698 | pub fn uid(&self) -> u32 {
| ^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `gid`
--> async_fuse/src/fuse_request.rs:704:5
|
704 | pub fn gid(&self) -> u32 {
| ^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `pid`
--> async_fuse/src/fuse_request.rs:710:5
|
710 | pub fn pid(&self) -> u32 {
| ^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `len`
--> async_fuse/src/fuse_request.rs:716:5
|
716 | pub fn len(&self) -> u32 {
| ^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (8/7)
--> fuse_ll/src/fuse/mod.rs:396:5
|
396 | / fn setxattr(
397 | | &mut self,
398 | | _req: &Request<'_>,
399 | | _ino: u64,
... |
404 | | reply: ReplyEmpty,
405 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: constant item is never used: `FOPEN_DIRECT_IO`
--> async_fuse/src/protocal.rs:152:1
|
152 | pub const FOPEN_DIRECT_IO: u32 = 1 << 0;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: constant item is never used: `FOPEN_KEEP_CACHE`
--> async_fuse/src/protocal.rs:153:1
|
153 | pub const FOPEN_KEEP_CACHE: u32 = 1 << 1;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: constant item is never used: `FUSE_POSIX_LOCKS`
--> async_fuse/src/protocal.rs:195:1
|
195 | pub const FUSE_POSIX_LOCKS: u32 = 1 << 1;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Fix following clippy warning
warning: field is never read: `uid`
--> fuse_ll/src/memfs.rs:893:5
|
893 | uid: Uid,
| ^^^^^^^^
|
= note: `#[warn(dead_code)]` on by default
warning: field is never read: `gid`
--> fuse_ll/src/memfs.rs:894:5
|
894 | gid: Gid,
| ^^^^^^^^
warning: unreachable pattern
--> async_fuse/src/fuse_request.rs:478:13
|
478 | _ => panic!("unknown FuseOpCode"),
| ^
|
= note: `#[warn(unreachable_patterns)]` on by default
warning: field is never read: `root_path`
--> fuse_ll/src/memfs.rs:895:5
|
895 | root_path: PathBuf,
| ^^^^^^^^^^^^^^^^^^
warning: method is never used: `new`
--> async_fuse/src/channel.rs:19:5
|
19 | pub async fn new(session: &Session) -> Result<Channel> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(dead_code)]` on by default
warning: method is never used: `fd`
--> async_fuse/src/channel.rs:51:5
|
51 | pub fn fd(&self) -> RawFd {
| ^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> fuse_ll/src/fuse/abi.rs:110:33
|
110 | pub const FATTR_MODE: u32 = 1 << 0;
| ^^^^^^
|
= note: `#[warn(clippy::identity_op)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `set_parent_ino`
--> async_fuse/src/fs/node.rs:62:5
|
62 | fn set_parent_ino(&mut self, parent: u64) -> INum {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `set_name`
--> async_fuse/src/fs/node.rs:71:5
|
71 | fn set_name(&mut self, name: OsString) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> fuse_ll/src/fuse/abi.rs:134:38
|
134 | pub const FOPEN_DIRECT_IO: u32 = 1 << 0; // bypass page cache for this open file
To use splice, sendfile, or mmap to support read zero copy as C libfuse does.
DatenLord master nodes and slave nodes should expose bunch of metrics for monitoring the performance and behavior.
The metric format should be compatible with Prometheus.
DatenLord may leverage Prometheus for monitoring and alerting.
For logging, it may use ELK or similar tools.
Fix following clippy warning
warning: method is never used: `statfs`
--> async_fuse/src/fuse_reply.rs:399:5
|
399 | / pub async fn statfs(
400 | | self,
401 | | blocks: u64,
402 | | bfree: u64,
... |
425 | | .await
426 | | }
| |_____^
warning: method is never used: `created`
--> async_fuse/src/fuse_reply.rs:446:5
|
446 | / pub async fn created(
447 | | self,
448 | | ttl: &Duration,
449 | | attr: FuseAttr,
... |
471 | | .await
472 | | }
| |_____^
warning: method is never used: `locked`
--> async_fuse/src/fuse_reply.rs:492:5
|
492 | pub async fn locked(self, start: u64, end: u64, typ: u32, pid: u32) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (15/7)
--> fuse_ll/src/fuse/mod.rs:129:5
|
129 | / fn setattr(
130 | | &mut self,
131 | | _req: &Request<'_>,
132 | | _ino: u64,
... |
144 | | reply: ReplyAttr,
145 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: method is never used: `bmap`
--> async_fuse/src/fuse_reply.rs:523:5
|
523 | pub async fn bmap(self, block: u64) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `error`
--> async_fuse/src/fuse_reply.rs:582:5
|
582 | pub async fn error(self, err: c_int) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `size`
--> async_fuse/src/fuse_reply.rs:599:5
|
599 | pub async fn size(self, size: u32) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (8/7)
--> fuse_ll/src/fuse/mod.rs:264:5
|
264 | / fn write(
265 | | &mut self,
266 | | _req: &Request<'_>,
267 | | _ino: u64,
... |
272 | | reply: ReplyWrite,
273 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: method is never used: `data`
--> async_fuse/src/fuse_reply.rs:606:5
|
606 | pub async fn data(self, bytes: Vec<u8>) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `len`
--> async_fuse/src/fuse_request.rs:17:5
|
17 | pub fn len(&self) -> usize {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
When processing FUSE requests concurrently, it's better to not share one FUSE device file handler but have multiple ones, so as to avoid the underlying lock associated with FUSE device..
So it might need to pre-open multiple FUSE device file handlers, and put them inside a pool to share.
To better understand performance bottlenecks , we have to use tracing tools to find out how DatenLord running on Linux.
Currently, Linux has two tracing mechanisms, perf and eBPF, and eBPF might be preferred.
For performance issue, we clone the device fd for each channel, one channel per Task, to avoid fd lock bottleneck.
It's part of the issue #1
Implement libfuse low level api using Rust async libraries, such as smol.
The libfuse low level api is defined here.
Fix following clippy warning
warning: 17 warnings emitted
warning: this function has too many arguments (8/7)
--> async_fuse/src/fs/mod.rs:1232:5
|
1232 | / pub async fn setxattr(
1233 | | &mut self,
1234 | | _req: &Request<'_>,
1235 | | _ino: u64,
... |
1240 | | reply: ReplyEmpty,
1241 | | ) -> anyhow::Result<()> {
| |___________________________^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: this function has too many arguments (10/7)
--> async_fuse/src/fs/mod.rs:1322:5
|
1322 | / pub async fn getlk(
1323 | | &mut self,
1324 | | _req: &Request<'_>,
1325 | | _ino: u64,
... |
1332 | | reply: ReplyLock,
1333 | | ) -> anyhow::Result<()> {
| |___________________________^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: constant item is never used: `FOPEN_DIRECT_IO`
--> fuse_ll/src/fuse/abi.rs:134:5
|
134 | pub const FOPEN_DIRECT_IO: u32 = 1 << 0; // bypass page cache for this open file
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(dead_code)]` on by default
warning: this function has too many arguments (11/7)
--> async_fuse/src/fs/mod.rs:1344:5
|
1344 | / pub async fn setlk(
1345 | | &mut self,
1346 | | _req: &Request<'_>,
1347 | | _ino: u64,
... |
1355 | | reply: ReplyEmpty,
1356 | | ) -> anyhow::Result<()> {
| |___________________________^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: constant item is never used: `FOPEN_KEEP_CACHE`
--> fuse_ll/src/fuse/abi.rs:135:5
|
135 | pub const FOPEN_KEEP_CACHE: u32 = 1 << 1; // don't invalidate the data cache on open
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (9/7)
--> async_fuse/src/fuse_reply.rs:399:5
|
399 | / pub async fn statfs(
400 | | self,
401 | | blocks: u64,
402 | | bfree: u64,
... |
408 | | frsize: u32,
409 | | ) -> anyhow::Result<()> {
| |___________________________^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: constant item is never used: `FUSE_POSIX_LOCKS`
--> fuse_ll/src/fuse/abi.rs:146:5
|
146 | pub const FUSE_POSIX_LOCKS: u32 = 1 << 1; // remote locking for POSIX file locks
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: constant item is never used: `FUSE_MIN_READ_BUFFER`
--> fuse_ll/src/fuse/abi.rs:227:5
|
227 | pub const FUSE_MIN_READ_BUFFER: usize = 8192;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: transmute from a pointer to a pointer
--> async_fuse/src/fuse_reply.rs:561:44
|
561 | let pdirent: *mut FuseDirEnt = mem::transmute(p);
| ^^^^^^^^^^^^^^^^^ help: try: `p as *mut protocal::FuseDirEnt`
|
= note: `#[warn(clippy::transmute_ptr_to_ptr)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#transmute_ptr_to_ptr
DatenLord needs master nodes to store all the metadata about the files on each slave node.
The master nodes might use RocksDB and Raft protocal to provide metadata service.
K8S defines its storage interface as CSI. If the container would like to access memfs, it's required to implement K8S CSI.
Master nodes need to manage some strategy about how to load, write data, such as:
if FUSE reply failed because of some IO error, the memfs will try to reply error to FUSE again, this is a bug need to fix
unreachable_pub lint warning is allowed by default, but it'll make pub keyword abused.
open an issue to track this lint warning.
once unreachable_pub is fixed, it'll be denied.
When using Rust async framework, the sequential style logging is no longer feasible.
It might leverage tracing framework from Tokio for async logging.
currently Channel::new() invokes some fcntl and ioctl functions from nix library, these functions are blocking calls, should be wrapped by smol::blocking!() to make them real async.
Fix following clippy warning
warning: this function has too many arguments (10/7)
--> fuse_ll/src/fuse/mod.rs:468:5
|
468 | / fn getlk(
469 | | &mut self,
470 | | _req: &Request<'_>,
471 | | _ino: u64,
... |
478 | | reply: ReplyLock,
479 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: this function has too many arguments (11/7)
--> fuse_ll/src/fuse/mod.rs:490:5
|
490 | / fn setlk(
491 | | &mut self,
492 | | _req: &Request<'_>,
493 | | _ino: u64,
... |
501 | | reply: ReplyEmpty,
502 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: using `clone` on a `Copy` type
--> fuse_ll/src/memfs.rs:123:30
|
123 | let st = stat::fstat(fd.clone())?;
| ^^^^^^^^^^ help: try removing the `clone` call: `fd`
|
= note: `#[warn(clippy::clone_on_copy)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
warning: this call to `map()` won't have an effect on the call to `count()`
--> fuse_ll/src/memfs.rs:486:27
|
486 | let entry_count = dir_node
| ___________________________^
487 | | .dir_fd
488 | | .borrow_mut()
489 | | .iter()
... |
519 | | })
520 | | .count();
| |____________________^
|
= note: `#[warn(clippy::suspicious_map)]` on by default
= help: make sure you did not confuse `map` with `filter` or `for_each`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#suspicious_map
We manage cache in the memfs, and the cache is supposed to be manipulated by multiple threads simultaneously. The cache manage system is used to handle parallel access. The lock-free data structure is one of the options. The finalized decision is made based on benchmark and profiling.
libfuse will negotiate will FUSE kernel module about bunch of capabilities when initializing.
These capabilities involves splice, flock, readdir_plus, etc.
CI will fail because of "error: could not find Cargo.toml
in /home/runner/work/datenlord/datenlord
or any parent directory"
Datenlord has two sub-projects async_fuse and fuse_ll. Cargo.toml exists in each folder instead of top folder.
implement the basic functionalities of async memfs based on async fuse
currently async_fuse only works on Ubuntu 18.04, it doesn't work on Ubuntu 20.04.
S3 will be used by a DatenLord slave node to load remote data from external device or sibling nodes, so it needs a S3 client.
DatenLord master nodes need to integrate with K8S scheduler, so that K8S can have data locality constrains when scheduling container jobs to make applications close to their data.
Basically, DatenLord will associate each slave node with one or more container job labels, and K8S will also label each container job with thease predefined labels, so as to mach the labels between jobs and nodes.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Some mount options are mutual exclusive, ex. "ro" and "rw". Need a check function for mount options.
Describe the solution you'd like
A clear and concise description of what you want to happen.
When users input mutual exclusive mount options, program will exit and report error.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Add a check function after getting all mount options.
Additional context
Add any other context or screenshots about the feature request here.
memfs should be out of async_fuse.
By doing so, async_fuse should define a FileSystem trait to expose FUSE operation methods to be implemented.
However, async_fuse and memfs internally use async functions, and currently Rust doesn't support defining async methods in a trait.
There is an external crate async_traits
can define async methods in a trait, maybe it's a workaround for now.
rename is the most complicate functionality of a filesystem, and implement it in async style.
the missing_docs lint warning is allowed by default, but it's not good for code health.
open an issue to track missing_docs warning.
Fix following clippy warning
warning: method is never used: `error`
--> fuse_ll/src/fuse/reply.rs:398:5
|
398 | pub fn error(self, err: c_int) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> async_fuse/src/protocal.rs:120:29
|
120 | pub const FATTR_MODE: u32 = 1 << 0;
| ^^^^^^
|
= note: `#[warn(clippy::identity_op)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `error`
--> fuse_ll/src/fuse/reply.rs:477:5
|
477 | pub fn error(self, err: c_int) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> async_fuse/src/protocal.rs:152:34
|
152 | pub const FOPEN_DIRECT_IO: u32 = 1 << 0;
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `created`
--> fuse_ll/src/fuse/reply.rs:500:5
|
500 | pub fn created(self, ttl: &Duration, attr: &FileAttr, generation: u64, fh: u64, flags: u32) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> async_fuse/src/protocal.rs:194:34
|
194 | pub const FUSE_ASYNC_READ: u32 = 1 << 0;
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `locked`
--> fuse_ll/src/fuse/reply.rs:543:5
|
543 | pub fn locked(self, start: u64, end: u64, typ: u32, pid: u32) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> async_fuse/src/protocal.rs:266:37
|
266 | pub const FUSE_RELEASE_FLUSH: u32 = 1 << 0;
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `bmap`
--> fuse_ll/src/fuse/reply.rs:578:5
|
578 | pub fn bmap(self, block: u64) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: 42 warnings emitted
DatenLord needs to support some well-known persistent storage systems, such as HDFS, NFS, Ceph, AWS S3, Aliyun OSS, etc, so that the date stored DatemLord can be exported to these external storage systems.
Before fixing all cargo clippy warnings, it should have integration tests to protect the code.
fuse_ll already have integration tests, so add to async fuse also.
Build async S3 server side API, maybe based on Rust async HTTP library TIDE.
Fix following clippy warning
warning: method is never used: `load_attribute`
--> async_fuse/src/fs/node.rs:123:5
|
123 | async fn load_attribute(&self) -> anyhow::Result<FileAttr> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> fuse_ll/src/fuse/abi.rs:145:38
|
145 | pub const FUSE_ASYNC_READ: u32 = 1 << 0; // asynchronous read requests
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `insert_entry`
--> async_fuse/src/fs/node.rs:447:5
|
447 | fn insert_entry(&mut self, child_entry: DirEntry) -> Option<DirEntry> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: the operation is ineffective. Consider reducing it to `1`
--> fuse_ll/src/fuse/abi.rs:186:41
|
186 | pub const FUSE_RELEASE_FLUSH: u32 = 1 << 0;
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op
warning: method is never used: `move_file`
--> async_fuse/src/fs/node.rs:658:5
|
658 | / fn move_file(
659 | | old_parent_node: &Node,
660 | | old_name: &OsStr,
661 | | new_parent_node: &Node,
... |
677 | | )
678 | | }
| |_____^
warning: method is never used: `with_capacity`
--> async_fuse/src/fuse_read.rs:20:5
|
20 | pub fn with_capacity(capacity: usize, reader: R) -> FuseBufReadStream<R> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `error`
--> async_fuse/src/fuse_reply.rs:358:5
|
358 | pub async fn error(self, err: c_int) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (9/7)
--> fuse_ll/src/fuse/reply.rs:449:5
|
449 | / pub fn statfs(
450 | | self,
451 | | blocks: u64,
452 | | bfree: u64,
... |
458 | | frsize: u32,
459 | | ) {
| |______^
|
= note: `#[warn(clippy::too_many_arguments)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: method is never used: `error`
--> async_fuse/src/fuse_reply.rs:382:5
|
382 | pub async fn error(self, err: c_int) -> anyhow::Result<()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: transmute from a pointer to a pointer
--> fuse_ll/src/fuse/reply.rs:619:45
|
619 | let pdirent: *mut fuse_dirent = mem::transmute(p);
| ^^^^^^^^^^^^^^^^^ help: try: `p as *mut fuse::abi::fuse_dirent`
|
= note: `#[warn(clippy::transmute_ptr_to_ptr)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#transmute_ptr_to_ptr
Support write zero copy using write_buf as C libfuse does.
Is your feature request related to a problem? Please describe.
There are lots of unsafe code when transforming struture to array in reply code. If we need to make it safe and efficient, a serializer is necessary.
Describe the solution you'd like
Implement the serializer for structure in FUSE friendly way.
Describe alternatives you've considered
N/A
Additional context
N/A
Currently there're many code style issues, we need to fix them and pass clippy. Add clippy check in github ci.
We should setup GitHub action to run some test and check while reviewing PR.
The check and test should involves cargo fmt, cargo test, cargo clippy, etc.
Fix following clippy warning
warning: constant item is never used: `FUSE_MIN_READ_BUFFER`
--> async_fuse/src/protocal.rs:426:1
|
426 | pub const FUSE_MIN_READ_BUFFER: usize = 8192;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (10/7)
--> fuse_ll/src/fuse/mod.rs:468:5
|
468 | / fn getlk(
469 | | &mut self,
470 | | _req: &Request<'_>,
471 | | _ino: u64,
... |
478 | | reply: ReplyLock,
479 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: constant item is never used: `FUSE_COMPAT_STATFS_SIZE`
--> async_fuse/src/protocal.rs:705:1
|
705 | pub const FUSE_COMPAT_STATFS_SIZE: usize = 48;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `fd`
--> async_fuse/src/session.rs:82:5
|
82 | pub fn fd(&self) -> RawFd {
| ^^^^^^^^^^^^^^^^^^^^^^^^^
warning: this function has too many arguments (11/7)
--> fuse_ll/src/fuse/mod.rs:490:5
|
490 | / fn setlk(
491 | | &mut self,
492 | | _req: &Request<'_>,
493 | | _ino: u64,
... |
501 | | reply: ReplyEmpty,
502 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: this function has too many arguments (15/7)
--> async_fuse/src/fs/mod.rs:529:5
|
529 | / pub async fn setattr(
530 | | &mut self,
531 | | req: &Request<'_>,
532 | | ino: u64,
... |
544 | | reply: ReplyAttr,
545 | | ) -> anyhow::Result<()> {
| |___________________________^
|
= note: `#[warn(clippy::too_many_arguments)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: using `clone` on a `Copy` type
--> fuse_ll/src/memfs.rs:123:30
|
123 | let st = stat::fstat(fd.clone())?;
| ^^^^^^^^^^ help: try removing the `clone` call: `fd`
|
= note: `#[warn(clippy::clone_on_copy)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
warning: this function has too many arguments (8/7)
--> async_fuse/src/fs/mod.rs:822:5
|
822 | / pub async fn write(
823 | | &mut self,
824 | | _req: &Request<'_>,
825 | | ino: u64,
... |
830 | | reply: ReplyWrite,
831 | | ) -> anyhow::Result<()> {
| |___________________________^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: this function has too many arguments (8/7)
--> async_fuse/src/fs/mod.rs:914:5
|
914 | / pub async fn release(
915 | | &mut self,
916 | | req: &Request<'_>,
917 | | ino: u64,
... |
922 | | reply: ReplyEmpty,
923 | | ) {
| |______^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
warning: this call to `map()` won't have an effect on the call to `count()`
--> fuse_ll/src/memfs.rs:486:27
|
486 | let entry_count = dir_node
| ___________________________^
487 | | .dir_fd
488 | | .borrow_mut()
489 | | .iter()
... |
519 | | })
520 | | .count();
| |____________________^
|
= note: `#[warn(clippy::suspicious_map)]` on by default
= help: make sure you did not confuse `map` with `filter` or `for_each`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#suspicious_map
The unsafe code is here:
datenlord/async_fuse/src/fuse_request.rs
Line 43 in 68b2988
An exploit:
use std::mem;
use anyhow::Context;
pub(crate) struct ByteSlice<'a> {
data: &'a [u8],
}
impl<'a> ByteSlice<'a>{
pub fn new(data: &'a [u8]) -> ByteSlice<'a> {
ByteSlice { data }
}
pub fn fetch_bytes(&mut self, amt: usize) -> anyhow::Result<&'a [u8]> {
if amt > self.data.len() {
return Err(anyhow::anyhow!(
"no enough bytes to fetch, remaining {} bytes but to fetch {} bytes",
self.data.len(),
amt,
));
}
let bytes = &self.data[..amt];
self.data = &self.data[amt..];
Ok(bytes)
}
pub fn fetch<T>(&mut self) -> anyhow::Result<&'a T> {
let len = mem::size_of::<T>();
let bytes = self.fetch_bytes(len).context(format!(
"failed to build FUSE request payload type {}",
std::any::type_name::<T>(),
))?;
let ret = unsafe { (bytes.as_ptr() as *const T).as_ref() };
match ret {
Some(obj) => Ok(obj),
None => Err(anyhow::anyhow!(
"failed to convert bytes to type={}",
std::any::type_name::<T>()
)),
}
}
}
fn main(){
use std::cell::Cell;
let buf: [u8; 128] = [0u8; 128];
{
// Thread A
let ref_a: &Cell<u128> = ByteSlice::new(&buf).fetch().unwrap();
ref_a.set(42);
}
{
// Thread B
let ref_b: &Cell<u128> = ByteSlice::new(&buf).fetch().unwrap();
ref_b.set(24); // potential race condition here
}
}
Run miri:
cargo +nightly miri
The output:
error: Undefined Behavior: accessing memory with alignment 1, but alignment 8 is required
--> /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/const_ptr.rs:100:57
|
100 | if self.is_null() { None } else { unsafe { Some(&*self) } }
| ^^^^^^ accessing memory with alignment 1, but alignment 8 is required
|
= help: this usually indicates that your program performed an invalid operation and caused Undefined Behavior
= help: but alignment errors can also be false positives, see https://github.com/rust-lang/miri/issues/1074
= help: you can disable the alignment check with `-Zmiri-disable-alignment-check`, but that could hide true bugs
= note: inside `std::ptr::const_ptr::<impl *const std::cell::Cell<u128>>::as_ref` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/const_ptr.rs:100:57
note: inside `ByteSlice::fetch::<std::cell::Cell<u128>>` at src/main.rs:32:28
--> src/main.rs:32:28
|
32 | let ret = unsafe { (bytes.as_ptr() as *const T).as_ref() };
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `main` at src/main.rs:50:34
--> src/main.rs:50:34
|
50 | let ref_a: &Cell<u128> = ByteSlice::new(&buf).fetch().unwrap();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: inside closure at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67:34
= note: inside closure at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:73
= note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<[closure@std::rt::lang_start_internal::{{closure}}#0::{{closure}}#0 0:&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/sys_common/backtrace.rs:130:5
= note: inside closure at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:13
= note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:348:40
= note: inside `std::panicking::r#try::<i32, [closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe]>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:325:15
= note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394:14
= note: inside `std::rt::lang_start_internal` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:51:25
= note: inside `std::rt::lang_start::<()>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67:5
error: aborting due to previous error
error: could not compile `testrs`.
To learn more, run the command again with --verbose.
The unsoundness has three aspects:
To avoid data copy when receiving FUSE request, it needs to hold a buffer for each FUSE request.
When handling FUSE request concurrently, it needs to have multiple buffers, each for one FUSE request exclusively.
When a FUSE request is finished processing, its buffer should be return back to the buffer pool to be reused for next FUSE request.
The buffer pool size can be defined as the number of max pending requests.
libfuse provides bunch of arguments which should be compatible with.
Fix following clippy warning
warning: method is never used: `size`
--> fuse_ll/src/fuse/reply.rs:663:5
|
663 | pub fn size(self, size: u32) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: method is never used: `data`
--> fuse_ll/src/fuse/reply.rs:668:5
|
668 | pub fn data(mut self, data: &[u8]) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: variant is never constructed: `NamedPipe`
--> fuse_ll/src/fuse/mod.rs:44:5
|
44 | NamedPipe,
| ^^^^^^^^^
warning: variant is never constructed: `CharDevice`
--> fuse_ll/src/fuse/mod.rs:46:5
|
46 | CharDevice,
| ^^^^^^^^^^
warning: variant is never constructed: `BlockDevice`
--> fuse_ll/src/fuse/mod.rs:48:5
|
48 | BlockDevice,
| ^^^^^^^^^^^
warning: variant is never constructed: `Symlink`
--> fuse_ll/src/fuse/mod.rs:54:5
|
54 | Symlink,
| ^^^^^^^
warning: variant is never constructed: `Socket`
--> fuse_ll/src/fuse/mod.rs:56:5
|
56 | Socket,
| ^^^^^^
warning: field is never read: `uid`
--> fuse_ll/src/memfs.rs:893:5
|
893 | uid: Uid,
| ^^^^^^^^
warning: field is never read: `gid`
--> fuse_ll/src/memfs.rs:894:5
|
894 | gid: Gid,
| ^^^^^^^^
warning: field is never read: `root_path`
--> fuse_ll/src/memfs.rs:895:5
|
895 | root_path: PathBuf,
| ^^^^^^^^^^^^^^^^^^
Advanced file operations includes setxattr, getxattr, listxattr, removexattr, getlk, setlk, flock, fallocation, readdirplus, copy_file_range, lseek, etc.
File system (FS) usually requires a journaling subsystem.
The FS journaling part is not like database binlog, journaling is not required to recover data, but binlog does.
The journaling design is not done yet.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.