Giter VIP home page Giter VIP logo

datenlord's Introduction

DatenLord

Join the chat at https://gitter.im/datenlord/datenlord codecov


DatenLord is a next-generation cloud-native distributed storage platform, which aims to meet the performance-critical storage needs from next-generation cloud-native applications, such as microservice, serverless, AI, etc. On one hand, DatenLord is designed to be a cloud-native storage system, which itself is distributed, fault-tolerant, and graceful upgrade. These cloud-native features make DatenLord easy to use and easy to maintain. On the other hand, DatenLord is designed as an application-orientated storage system, in that DatenLord is optimized for many performance-critical scenarios, such as databases, AI machine learning, big data. Meanwhile, DatenLord provides high-performance storage service for containers, which facilitates stateful applications running on top of Kubernetes (K8S). The high performance of DatenLord is achieved by leveraging the most recent technology revolution in hardware and software, such as NVMe, non-volatile memory, asynchronous programming, and the native Linux asynchronous IO support.


Why DatenLord?

Why do we build DatenLord? The reason is two-fold:

  • Firstly, the recent computer hardware architecture revolution stimulates storage software refractory. The storage related functionalities inside Linux kernel haven't changed much in recent 10 years, whenas hard-disk drive (HDD) was the main storage device. Nowadays, solid-state drive (SSD) becomes the mainstream, not even mention the most advanced SSD, NVMe and non-volatile memory. The performance of SSD is hundreds of times faster than HDD, in that the HDD latency is around 1~10 ms, whereas the SSD latency is around 50–150 μs, the NVMe latency is around 25 μs, and the non-volatile memory latency is 350 ns. With the performance revolution of storage devices, traditional blocking-style/synchronous IO in Linux kernel becomes very inefficient, and non-blocking-style/asynchronous IO is much more applicable. The Linux kernel community already realized that, and recently Linux kernel has proposed native-asynchronous IO mechanism, io_uring, to improve IO performance. Beside blocking-style/synchronous IO, the context switch overhead in Linux kernel becomes no longer negligible w.r.t. SSD latency. Many modern programming languages have proposed asynchronous programming, green thread or coroutine to manage asynchronous IO tasks in user space, in order to avoid context switch overhead introduced by blocking IO. Therefore we think it’s time to build a next-generation storage system that takes advantage of the storage performance revolution as far as possible, by leveraging non-blocking/asynchronous IO, asynchronous programming, NVMe, and even non-volatile memory, etc.

  • Secondly, most distributed/cloud-native systems are computing and storage isolated, that computing tasks/applications and storage systems are of dedicated clusters, respectively. This isolated architecture is best to reduce maintenance, that it decouples the maintenance tasks of computing clusters and storage clusters into separate ones, such as upgrade, expansion, migration of each cluster respectively, which is much simpler than of coupled clusters. Nowadays, however, applications are dealing with much larger datasets than ever before. One notorious example is that an AI training job takes one hour to load data whereas the training job itself finishes in only 45 minutes. Therefore, isolating computing and storage makes IO very inefficient, as transferring data between applications and storage systems via network takes quite a lot of time. Further, with the isolated architecture, applications have to be aware of the different data location, and the varying access cost due to the difference of data location, network distance, etc. DatenLord tackles the IO performance issue of isolated architecture in a novel way, which abstracts the heterogeneous storage details and makes the difference of data location, access cost, etc, transparent to applications. Furthermore, with DatenLord, applications can assume all the data to be accessed are local, and DatenLord will access the data on behalf of applications. Besides, DatenLord can help K8S to schedule jobs close to cached data, since DatenLord knows the exact location of all cached data. By doing so, applications are greatly simplified w.r.t. to data access, and DatenLord can leverage local cache, neighbor cache, and remote cache to speed up data access, so as to boost performance.


Target scenarios

The main scenario of DatenLord is to facilitate high availability across multi-cloud, hybrid-cloud, multiple data centers, etc. Concretely, there are many online business providers whose business is too important to afford any downtime. To achieve high availability, the service providers have to leverage multi-cloud, hybrid-cloud, and multiple data centers to hopefully avoid single point failure of each single cloud or data center, by deploying applications and services across multiple clouds or data centers. It's relatively easier to deploy applications and services to multiple clouds and data centers, but it's much harder to duplicate all data to all clouds or all data centers in a timely manner, due to the huge data size. If data is not equally available across multiple clouds or data centers, the online business might still suffer from single point failure of a cloud or a data center, because data unavailability resulted from a cloud or a data center failure.

DatenLord can alleviate data unavailable of cloud or data center failure by caching data to multiple layers, such as local cache, neighbor cache, remote cache, etc. Although the total data size is huge, the hot data involved in online business is usually of limited size, which is called data locality. DatenLord leverages data locality and builds a set of large scale distributed and automatic cache layers to buffer hot data in a smart manner. The benefit of DatenLord is two-fold:

  • DatenLord is transparent to applications, namely DatenLord does not need any modification to applications;
  • DatenLord is high performance, that it automatically caches data by means of the data hotness, and it's performance is achieved by applying different caching strategies according to target applications. For example, least recent use (LRU) caching strategy for some kind of random access, most recent use (MRU) caching strategy for some kind of sequential access, etc.

Architecture

Single Data Center

DatenLord Single Data Center

Multiple Data Centers and Hybrid Cloud

DatenLord Multiple Data Centers and Hybrid Cloud

DatenLord provides 3 kinds of user interfaces: KV interface, S3 interface and file interface. The backend storage is supported by the underlying distributed cache layer which is strong consistent. The strong consistency is guaranteed by the metadata management module which is built on high performance consensus protocol. The persistence storage layer can be local disk or S3 storage. For the network, RDMA is used to provide high throughput and low latency networks. If RDMA is not supported, TCP is an alternative option. For the multiple data center and hybrid clouds scenario, there will be a dedicated metadata server which supports metadata requests within the same data center. While in the same data center scenario, the metadata module can run on the same machine as the cache node. The network between data centers and public clouds are managed by a private network to guarantee high quality data transfer.

Quick Start

Currently DatenLord has been built as Docker images and can be deployed via K8S.

To deploy DatenLord via K8S, just simply run:

  • sed -e 's/e2e_test/latest/g' scripts/setup/datenlord.yaml > datenlord-deploy.yaml
  • kubectl apply -f datenlord-deploy.yaml

To use DatenLord, just define PVC using DatenLord Storage Class, and then deploy a Pod using this PVC:

cat <<EOF >datenlord-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-datenlord-test
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 100Mi
  storageClassName: csi-datenlord-sc

---
apiVersion: v1
kind: Pod
metadata:
  name: mysql-datenlord-test
spec:
    containers:
    - name: mysql
      image: mysql
      env:
      - name: MYSQL_ROOT_PASSWORD
        value: "rootpasswd"
      volumeMounts:
      - mountPath: /var/lib/mysql
        name: data
        subPath: mysql
    volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pvc-datenlord-test
EOF

kubectl apply -f datenlord-demo.yaml

DatenLord provides a customized scheduler which implements K8S scheduler extender. The scheduler will try to schedule a pod to the node that has the volume that it requests. To use the scheduler, add schedulerName: datenlord-scheduler to the spec of your pod. Caveat: dangling docker image may cause failed to parse request error. Doing docker image prune on each K8S node is a way to fix it.

It may need to install snapshot CRD and controller on K8S, if used K8S CSI snapshot feature:

  • kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
  • kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
  • kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
  • kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
  • kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Monitoring

Datenlord monitoring guideline is in datenlord monitoring. We provide both YAML and Helm method to deploy the monitoring system.

To use YAML method, just run

sh ./scripts/setup/datenlord-monitor-deploy.sh

To use Helm method, run

sh ./scripts/setup/datenlord-monitor-deploy.sh helm

Performance Test

Performance test is done by fio and fio-plot is used to plot performance histograms.

To run performance test,

sudo apt-get update
sudo apt-get install -y fio python3-pip
sudo pip3 install matplotlib numpy fio-plot
sh ./scripts/perf/fio-perf-test.sh TEST_DIR

Four histograms will be generated.

  • Random read IOPS and latency for different block sizes
  • Random write IOPS and latency for different block sizes
  • Random read IOPS and latency for different read thread numbers with 4k block size
  • Random write IOPS and latency for different write thread numbers with 4k block size

Performance test is added to GitHub Action(cron.yml) and performance report is generated and archived as artifacts(Example) for every four hours.

How to Contribute

Anyone interested in DatenLord is welcomed to contribute.

Coding Style

Please follow the code style. Meanwhile, DatenLord adopts very strict clippy linting, please fix every clippy warning before submit your PR. Also please make sure all CI tests are passed.

Continuous Integration (CI)

The CI of DatenLord leverages GitHub Action. There are two CI flows for DatenLord, One is for Rust cargo test, clippy lints, and standard filesystem E2E checks; The other is for CSI related tests, such as CSI sanity test and CSI E2E test.

The CSI E2E test setup is a bit complex, its action script cron.yml is quite long, so let's explain it in detail:

  • First, it sets up a test K8S cluster with one master node and three slave nodes, using Kubernetes in Docker (KinD);

  • Second, CSI E2E test requires no-password SSH login to each K8S slave node, since it might run some commands to prepare test environment or verify test result, so it has to setup SSH key to each Docker container of KinD slave nodes;

  • Third, it builds DatenLord container images and loads to KinD, which is a caveat of KinD, in that KinD puts K8S nodes inside Docker containers, thus kubelet cannot reach any resource of local host, and KinD provides load operation to make the container images from local host visible to kubelet;

  • At last, it deploys DatenLord to the test K8S cluster, then downloads pre-build K8S E2E binary, runs in parallel by involking ginkgo -p, and only selects External.Storage related CSI E2E testcases to run.

Sub-Projects

DatenLord has several related sub-projects, mostly working in progress, listed alphabetically:

Road Map

  • 0.1 Refactor async fuse lib to provide clear async APIs, which is used by the datenlord filesystem.
  • 0.2 Support all Fuse APIs in the datenlord fs.
  • 0.3 Make fuse lib fully asynchronous. Switch async fuse lib's device communication channel from blocking I/O to io_uring.
  • 0.4 Complete K8S integration test.
  • 0.5 Support RDMA.
  • 1.0 Complete Tensorflow K8S integration and finish performance comparison with raw fs.

datenlord's People

Contributors

activepeter avatar anarion-zuo avatar dependabot-preview[bot] avatar dependabot[bot] avatar devillove084 avatar genedna avatar gitter-badger avatar gtwhy avatar ironicbo avatar lord2anil avatar mrwwheat7 avatar name1e5s avatar nugine avatar pwang7 avatar pymongo avatar pzheng1025 avatar reto911 avatar rogercloud avatar u5surf avatar uran0sh avatar xiaguan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

datenlord's Issues

Clippy Fix 1

Fix following clippy warning

warning: field is never read: `uid`
   --> fuse_ll/src/memfs.rs:893:5
    |
893 |     uid: Uid,
    |     ^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: field is never read: `gid`
   --> fuse_ll/src/memfs.rs:894:5
    |
894 |     gid: Gid,
    |     ^^^^^^^^

warning: unreachable pattern
   --> async_fuse/src/fuse_request.rs:478:13
    |
478 |             _ => panic!("unknown FuseOpCode"),
    |             ^
    |
    = note: `#[warn(unreachable_patterns)]` on by default

warning: field is never read: `root_path`
   --> fuse_ll/src/memfs.rs:895:5
    |
895 |     root_path: PathBuf,
    |     ^^^^^^^^^^^^^^^^^^

warning: method is never used: `new`
  --> async_fuse/src/channel.rs:19:5
   |
19 |     pub async fn new(session: &Session) -> Result<Channel> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: method is never used: `fd`
  --> async_fuse/src/channel.rs:51:5
   |
51 |     pub fn fd(&self) -> RawFd {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> fuse_ll/src/fuse/abi.rs:110:33
    |
110 |     pub const FATTR_MODE: u32 = 1 << 0;
    |                                 ^^^^^^
    |
    = note: `#[warn(clippy::identity_op)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `set_parent_ino`
  --> async_fuse/src/fs/node.rs:62:5
   |
62 |     fn set_parent_ino(&mut self, parent: u64) -> INum {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `set_name`
  --> async_fuse/src/fs/node.rs:71:5
   |
71 |     fn set_name(&mut self, name: OsString) {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> fuse_ll/src/fuse/abi.rs:134:38
    |
134 |     pub const FOPEN_DIRECT_IO: u32 = 1 << 0; // bypass page cache for this open file

Support external persistent storage systems

DatenLord needs to support some well-known persistent storage systems, such as HDFS, NFS, Ceph, AWS S3, Aliyun OSS, etc, so that the date stored DatemLord can be exported to these external storage systems.

Clippy Fix 5

Fix following clippy warning

warning: constant item is never used: `FUSE_MIN_READ_BUFFER`
   --> async_fuse/src/protocal.rs:426:1
    |
426 | pub const FUSE_MIN_READ_BUFFER: usize = 8192;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (10/7)
   --> fuse_ll/src/fuse/mod.rs:468:5
    |
468 | /     fn getlk(
469 | |         &mut self,
470 | |         _req: &Request<'_>,
471 | |         _ino: u64,
...   |
478 | |         reply: ReplyLock,
479 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: constant item is never used: `FUSE_COMPAT_STATFS_SIZE`
   --> async_fuse/src/protocal.rs:705:1
    |
705 | pub const FUSE_COMPAT_STATFS_SIZE: usize = 48;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `fd`
  --> async_fuse/src/session.rs:82:5
   |
82 |     pub fn fd(&self) -> RawFd {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (11/7)
   --> fuse_ll/src/fuse/mod.rs:490:5
    |
490 | /     fn setlk(
491 | |         &mut self,
492 | |         _req: &Request<'_>,
493 | |         _ino: u64,
...   |
501 | |         reply: ReplyEmpty,
502 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: this function has too many arguments (15/7)
   --> async_fuse/src/fs/mod.rs:529:5
    |
529 | /     pub async fn setattr(
530 | |         &mut self,
531 | |         req: &Request<'_>,
532 | |         ino: u64,
...   |
544 | |         reply: ReplyAttr,
545 | |     ) -> anyhow::Result<()> {
    | |___________________________^
    |
    = note: `#[warn(clippy::too_many_arguments)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: using `clone` on a `Copy` type
   --> fuse_ll/src/memfs.rs:123:30
    |
123 |         let st = stat::fstat(fd.clone())?;
    |                              ^^^^^^^^^^ help: try removing the `clone` call: `fd`
    |
    = note: `#[warn(clippy::clone_on_copy)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy

warning: this function has too many arguments (8/7)
   --> async_fuse/src/fs/mod.rs:822:5
    |
822 | /     pub async fn write(
823 | |         &mut self,
824 | |         _req: &Request<'_>,
825 | |         ino: u64,
...   |
830 | |         reply: ReplyWrite,
831 | |     ) -> anyhow::Result<()> {
    | |___________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: this function has too many arguments (8/7)
   --> async_fuse/src/fs/mod.rs:914:5
    |
914 | /     pub async fn release(
915 | |         &mut self,
916 | |         req: &Request<'_>,
917 | |         ino: u64,
...   |
922 | |         reply: ReplyEmpty,
923 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: this call to `map()` won't have an effect on the call to `count()`
   --> fuse_ll/src/memfs.rs:486:27
    |
486 |           let entry_count = dir_node
    |  ___________________________^
487 | |             .dir_fd
488 | |             .borrow_mut()
489 | |             .iter()
...   |
519 | |             })
520 | |             .count();
    | |____________________^
    |
    = note: `#[warn(clippy::suspicious_map)]` on by default
    = help: make sure you did not confuse `map` with `filter` or `for_each`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#suspicious_map

Handle FUSE reply failure

if FUSE reply failed because of some IO error, the memfs will try to reply error to FUSE again, this is a bug need to fix

Clippy Fix 6

Fix following clippy warning

warning: 17 warnings emitted

warning: this function has too many arguments (8/7)
    --> async_fuse/src/fs/mod.rs:1232:5
     |
1232 | /     pub async fn setxattr(
1233 | |         &mut self,
1234 | |         _req: &Request<'_>,
1235 | |         _ino: u64,
...    |
1240 | |         reply: ReplyEmpty,
1241 | |     ) -> anyhow::Result<()> {
     | |___________________________^
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: this function has too many arguments (10/7)
    --> async_fuse/src/fs/mod.rs:1322:5
     |
1322 | /     pub async fn getlk(
1323 | |         &mut self,
1324 | |         _req: &Request<'_>,
1325 | |         _ino: u64,
...    |
1332 | |         reply: ReplyLock,
1333 | |     ) -> anyhow::Result<()> {
     | |___________________________^
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: constant item is never used: `FOPEN_DIRECT_IO`
   --> fuse_ll/src/fuse/abi.rs:134:5
    |
134 |     pub const FOPEN_DIRECT_IO: u32 = 1 << 0; // bypass page cache for this open file
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: this function has too many arguments (11/7)
    --> async_fuse/src/fs/mod.rs:1344:5
     |
1344 | /     pub async fn setlk(
1345 | |         &mut self,
1346 | |         _req: &Request<'_>,
1347 | |         _ino: u64,
...    |
1355 | |         reply: ReplyEmpty,
1356 | |     ) -> anyhow::Result<()> {
     | |___________________________^
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: constant item is never used: `FOPEN_KEEP_CACHE`
   --> fuse_ll/src/fuse/abi.rs:135:5
    |
135 |     pub const FOPEN_KEEP_CACHE: u32 = 1 << 1; // don't invalidate the data cache on open
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (9/7)
   --> async_fuse/src/fuse_reply.rs:399:5
    |
399 | /     pub async fn statfs(
400 | |         self,
401 | |         blocks: u64,
402 | |         bfree: u64,
...   |
408 | |         frsize: u32,
409 | |     ) -> anyhow::Result<()> {
    | |___________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: constant item is never used: `FUSE_POSIX_LOCKS`
   --> fuse_ll/src/fuse/abi.rs:146:5
    |
146 |     pub const FUSE_POSIX_LOCKS: u32 = 1 << 1; // remote locking for POSIX file locks
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: constant item is never used: `FUSE_MIN_READ_BUFFER`
   --> fuse_ll/src/fuse/abi.rs:227:5
    |
227 |     pub const FUSE_MIN_READ_BUFFER: usize = 8192;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: transmute from a pointer to a pointer
   --> async_fuse/src/fuse_reply.rs:561:44
    |
561 |             let pdirent: *mut FuseDirEnt = mem::transmute(p);
    |                                            ^^^^^^^^^^^^^^^^^ help: try: `p as *mut protocal::FuseDirEnt`
    |
    = note: `#[warn(clippy::transmute_ptr_to_ptr)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#transmute_ptr_to_ptr

Unsoundness in ByteSlice::fetch

The unsafe code is here:

let ret = unsafe { (bytes.as_ptr() as *const T).as_ref() };

An exploit:

use std::mem;
use anyhow::Context;

pub(crate) struct ByteSlice<'a> {
    data: &'a [u8],
}

impl<'a> ByteSlice<'a>{
    pub fn new(data: &'a [u8]) -> ByteSlice<'a> {
        ByteSlice { data }
    }

    pub fn fetch_bytes(&mut self, amt: usize) -> anyhow::Result<&'a [u8]> {
        if amt > self.data.len() {
            return Err(anyhow::anyhow!(
                "no enough bytes to fetch, remaining {} bytes but to fetch {} bytes",
                self.data.len(),
                amt,
            ));
        }
        let bytes = &self.data[..amt];
        self.data = &self.data[amt..];
        Ok(bytes)
    }

    pub fn fetch<T>(&mut self) -> anyhow::Result<&'a T> {
        let len = mem::size_of::<T>();
        let bytes = self.fetch_bytes(len).context(format!(
            "failed to build FUSE request payload type {}",
            std::any::type_name::<T>(),
        ))?;
        let ret = unsafe { (bytes.as_ptr() as *const T).as_ref() };
        match ret {
            Some(obj) => Ok(obj),
            None => Err(anyhow::anyhow!(
                "failed to convert bytes to type={}",
                std::any::type_name::<T>()
            )),
        }
    }
}

fn main(){
    use std::cell::Cell;

    let buf: [u8; 128] = [0u8; 128];

    {
        // Thread A
        let ref_a: &Cell<u128> = ByteSlice::new(&buf).fetch().unwrap();
        ref_a.set(42);
    }
    {
        // Thread B
        let ref_b: &Cell<u128> = ByteSlice::new(&buf).fetch().unwrap();
        ref_b.set(24); // potential race condition here
    }
}

Run miri:

cargo +nightly miri

The output:

error: Undefined Behavior: accessing memory with alignment 1, but alignment 8 is required
   --> /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/const_ptr.rs:100:57
    |
100 |         if self.is_null() { None } else { unsafe { Some(&*self) } }
    |                                                         ^^^^^^ accessing memory with alignment 1, but alignment 8 is required
    |
    = help: this usually indicates that your program performed an invalid operation and caused Undefined Behavior
    = help: but alignment errors can also be false positives, see https://github.com/rust-lang/miri/issues/1074
    = help: you can disable the alignment check with `-Zmiri-disable-alignment-check`, but that could hide true bugs
            
    = note: inside `std::ptr::const_ptr::<impl *const std::cell::Cell<u128>>::as_ref` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/const_ptr.rs:100:57
note: inside `ByteSlice::fetch::<std::cell::Cell<u128>>` at src/main.rs:32:28
   --> src/main.rs:32:28
    |
32  |         let ret = unsafe { (bytes.as_ptr() as *const T).as_ref() };
    |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `main` at src/main.rs:50:34
   --> src/main.rs:50:34
    |
50  |         let ref_a: &Cell<u128> = ByteSlice::new(&buf).fetch().unwrap();
    |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = note: inside closure at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67:34
    = note: inside closure at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:73
    = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<[closure@std::rt::lang_start_internal::{{closure}}#0::{{closure}}#0 0:&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/sys_common/backtrace.rs:130:5
    = note: inside closure at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:13
    = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:348:40
    = note: inside `std::panicking::r#try::<i32, [closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe]>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:325:15
    = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{{closure}}#0 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394:14
    = note: inside `std::rt::lang_start_internal` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:51:25
    = note: inside `std::rt::lang_start::<()>` at /home/nugine/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67:5

error: aborting due to previous error

error: could not compile `testrs`.

To learn more, run the command again with --verbose.

The unsoundness has three aspects:

  1. Unconditional transmutation
  2. Unsync access
  3. Alignment error (a false positive?)

Enable unreachable_pub lint

unreachable_pub lint warning is allowed by default, but it'll make pub keyword abused.
open an issue to track this lint warning.
once unreachable_pub is fixed, it'll be denied.

Data manager on master nodes

Master nodes need to manage some strategy about how to load, write data, such as:

  1. Preload remote data for AI training jobs;
  2. Asynchronously back up write data in realtime to local filesystem or remote storage systems;
  3. Replica control, especially when some nodes are down;
  4. Rebalance data across slave nodes, without affecting storage traffic.

Clippy Fix 4

Fix following clippy warning

warning: method is never used: `fetch_slice`
  --> async_fuse/src/fuse_request.rs:52:5
   |
52 |     pub fn fetch_slice<T>(&mut self) -> anyhow::Result<Vec<&'a T>> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (8/7)
   --> fuse_ll/src/fuse/mod.rs:306:5
    |
306 | /     fn release(
307 | |         &mut self,
308 | |         _req: &Request<'_>,
309 | |         _ino: u64,
...   |
314 | |         reply: ReplyEmpty,
315 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: method is never used: `uid`
   --> async_fuse/src/fuse_request.rs:698:5
    |
698 |     pub fn uid(&self) -> u32 {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `gid`
   --> async_fuse/src/fuse_request.rs:704:5
    |
704 |     pub fn gid(&self) -> u32 {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `pid`
   --> async_fuse/src/fuse_request.rs:710:5
    |
710 |     pub fn pid(&self) -> u32 {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `len`
   --> async_fuse/src/fuse_request.rs:716:5
    |
716 |     pub fn len(&self) -> u32 {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (8/7)
   --> fuse_ll/src/fuse/mod.rs:396:5
    |
396 | /     fn setxattr(
397 | |         &mut self,
398 | |         _req: &Request<'_>,
399 | |         _ino: u64,
...   |
404 | |         reply: ReplyEmpty,
405 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: constant item is never used: `FOPEN_DIRECT_IO`
   --> async_fuse/src/protocal.rs:152:1
    |
152 | pub const FOPEN_DIRECT_IO: u32 = 1 << 0;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: constant item is never used: `FOPEN_KEEP_CACHE`
   --> async_fuse/src/protocal.rs:153:1
    |
153 | pub const FOPEN_KEEP_CACHE: u32 = 1 << 1;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: constant item is never used: `FUSE_POSIX_LOCKS`
   --> async_fuse/src/protocal.rs:195:1
    |
195 | pub const FUSE_POSIX_LOCKS: u32 = 1 << 1;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Add integration test for async fuse

Before fixing all cargo clippy warnings, it should have integration tests to protect the code.
fuse_ll already have integration tests, so add to async fuse also.

fuse_ll: Add mutual exclusive check for mount options

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Some mount options are mutual exclusive, ex. "ro" and "rw". Need a check function for mount options.

Describe the solution you'd like
A clear and concise description of what you want to happen.
When users input mutual exclusive mount options, program will exit and report error.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Add a check function after getting all mount options.

Additional context
Add any other context or screenshots about the feature request here.

Async S3 client side

S3 will be used by a DatenLord slave node to load remote data from external device or sibling nodes, so it needs a S3 client.

Clippy Fix 2

Fix following clippy warning

warning: method is never used: `load_attribute`
   --> async_fuse/src/fs/node.rs:123:5
    |
123 |     async fn load_attribute(&self) -> anyhow::Result<FileAttr> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> fuse_ll/src/fuse/abi.rs:145:38
    |
145 |     pub const FUSE_ASYNC_READ: u32 = 1 << 0; // asynchronous read requests
    |                                      ^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `insert_entry`
   --> async_fuse/src/fs/node.rs:447:5
    |
447 |     fn insert_entry(&mut self, child_entry: DirEntry) -> Option<DirEntry> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> fuse_ll/src/fuse/abi.rs:186:41
    |
186 |     pub const FUSE_RELEASE_FLUSH: u32 = 1 << 0;
    |                                         ^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `move_file`
   --> async_fuse/src/fs/node.rs:658:5
    |
658 | /     fn move_file(
659 | |         old_parent_node: &Node,
660 | |         old_name: &OsStr,
661 | |         new_parent_node: &Node,
...   |
677 | |         )
678 | |     }
    | |_____^

warning: method is never used: `with_capacity`
  --> async_fuse/src/fuse_read.rs:20:5
   |
20 |     pub fn with_capacity(capacity: usize, reader: R) -> FuseBufReadStream<R> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `error`
   --> async_fuse/src/fuse_reply.rs:358:5
    |
358 |     pub async fn error(self, err: c_int) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (9/7)
   --> fuse_ll/src/fuse/reply.rs:449:5
    |
449 | /     pub fn statfs(
450 | |         self,
451 | |         blocks: u64,
452 | |         bfree: u64,
...   |
458 | |         frsize: u32,
459 | |     ) {
    | |______^
    |
    = note: `#[warn(clippy::too_many_arguments)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: method is never used: `error`
   --> async_fuse/src/fuse_reply.rs:382:5
    |
382 |     pub async fn error(self, err: c_int) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: transmute from a pointer to a pointer
   --> fuse_ll/src/fuse/reply.rs:619:45
    |
619 |             let pdirent: *mut fuse_dirent = mem::transmute(p);
    |                                             ^^^^^^^^^^^^^^^^^ help: try: `p as *mut fuse::abi::fuse_dirent`
    |
    = note: `#[warn(clippy::transmute_ptr_to_ptr)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#transmute_ptr_to_ptr

Implement rename

rename is the most complicate functionality of a filesystem, and implement it in async style.

Missing docs

the missing_docs lint warning is allowed by default, but it's not good for code health.
open an issue to track missing_docs warning.

Monitoring metrics

DatenLord master nodes and slave nodes should expose bunch of metrics for monitoring the performance and behavior.

The metric format should be compatible with Prometheus.

File System Journaling

File system (FS) usually requires a journaling subsystem.
The FS journaling part is not like database binlog, journaling is not required to recover data, but binlog does.

The journaling design is not done yet.

Enable trivial_casts lint

there are several trivial casts in the code, they have to be fixed before enabling trivial_casts lint.

error: trivial cast: `&T` as `*const T`
  --> async_fuse/src/fuse_reply.rs:72:37
   |
72 | ...                   let p = &instance as *const T as *const u8;
   |                               ^^^^^^^^^^^^^^^^^^^^^
   |
note: the lint level is defined here
  --> async_fuse/src/main.rs:12:5
   |
12 |     trivial_casts, //TODO: remove trivial casts in code
   |     ^^^^^^^^^^^^^
   = help: cast can be replaced by coercion; this might require a temporary variable

error: trivial cast: `&protocol::FuseOutHeader` as `*const protocol::FuseOutHeader`
  --> async_fuse/src/fuse_reply.rs:94:21
   |
94 |             let h = &header as *const FuseOutHeader as *const u8;
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: cast can be replaced by coercion; this might require a temporary variable

error: trivial cast: `&std::ffi::OsStr` as `*const std::ffi::OsStr`
   --> async_fuse/src/mount.rs:255:42
    |
255 |         let res = unsafe { libc::unmount(mntpnt as *const _ as *const u8 as *const i8, MNT_FORCE) };
    |                                          ^^^^^^^^^^^^^^^^^^
    |
    = help: cast can be replaced by coercion; this might require a temporary variable

error: trivial cast: `&mut u32` as `*mut u32`
   --> async_fuse/src/mount.rs:281:52
    |
281 |         let result = unsafe { fuse_read_random(fd, &mut drandom as *mut _)? };
    |                                                    ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: cast can be replaced by coercion; this might require a temporary variable

error: trivial cast: `&mut mount::param::FuseMountArgs` as `*mut mount::param::FuseMountArgs`
   --> async_fuse/src/mount.rs:346:17
    |
346 |                 &mut mnt_args as *mut _ as *mut c_void,
    |                 ^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: cast can be replaced by coercion; this might require a temporary variable

error: aborting due to 5 previous errors

Lock-free cache manage system

We manage cache in the memfs, and the cache is supposed to be manipulated by multiple threads simultaneously. The cache manage system is used to handle parallel access. The lock-free data structure is one of the options. The finalized decision is made based on benchmark and profiling.

K8S scheduler integration

DatenLord master nodes need to integrate with K8S scheduler, so that K8S can have data locality constrains when scheduling container jobs to make applications close to their data.

Basically, DatenLord will associate each slave node with one or more container job labels, and K8S will also label each container job with thease predefined labels, so as to mach the labels between jobs and nodes.

eBPF tracing and perf

To better understand performance bottlenecks , we have to use tracing tools to find out how DatenLord running on Linux.
Currently, Linux has two tracing mechanisms, perf and eBPF, and eBPF might be preferred.

Enable unsafe_code lint

to enable unsafe_code lint, we have to remove unsafe code, or only allow minimum unsafe code for performance consideration.

Make Channel real async

currently Channel::new() invokes some fcntl and ioctl functions from nix library, these functions are blocking calls, should be wrapped by smol::blocking!() to make them real async.

Clippy Fix 9

Fix following clippy warning

warning: this function has too many arguments (10/7)
   --> fuse_ll/src/fuse/mod.rs:468:5
    |
468 | /     fn getlk(
469 | |         &mut self,
470 | |         _req: &Request<'_>,
471 | |         _ino: u64,
...   |
478 | |         reply: ReplyLock,
479 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: this function has too many arguments (11/7)
   --> fuse_ll/src/fuse/mod.rs:490:5
    |
490 | /     fn setlk(
491 | |         &mut self,
492 | |         _req: &Request<'_>,
493 | |         _ino: u64,
...   |
501 | |         reply: ReplyEmpty,
502 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: using `clone` on a `Copy` type
   --> fuse_ll/src/memfs.rs:123:30
    |
123 |         let st = stat::fstat(fd.clone())?;
    |                              ^^^^^^^^^^ help: try removing the `clone` call: `fd`
    |
    = note: `#[warn(clippy::clone_on_copy)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy

warning: this call to `map()` won't have an effect on the call to `count()`
   --> fuse_ll/src/memfs.rs:486:27
    |
486 |           let entry_count = dir_node
    |  ___________________________^
487 | |             .dir_fd
488 | |             .borrow_mut()
489 | |             .iter()
...   |
519 | |             })
520 | |             .count();
    | |____________________^
    |
    = note: `#[warn(clippy::suspicious_map)]` on by default
    = help: make sure you did not confuse `map` with `filter` or `for_each`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#suspicious_map

Clippy Fix 3

Fix following clippy warning

warning: method is never used: `statfs`
   --> async_fuse/src/fuse_reply.rs:399:5
    |
399 | /     pub async fn statfs(
400 | |         self,
401 | |         blocks: u64,
402 | |         bfree: u64,
...   |
425 | |             .await
426 | |     }
    | |_____^

warning: method is never used: `created`
   --> async_fuse/src/fuse_reply.rs:446:5
    |
446 | /     pub async fn created(
447 | |         self,
448 | |         ttl: &Duration,
449 | |         attr: FuseAttr,
...   |
471 | |             .await
472 | |     }
    | |_____^

warning: method is never used: `locked`
   --> async_fuse/src/fuse_reply.rs:492:5
    |
492 |     pub async fn locked(self, start: u64, end: u64, typ: u32, pid: u32) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (15/7)
   --> fuse_ll/src/fuse/mod.rs:129:5
    |
129 | /     fn setattr(
130 | |         &mut self,
131 | |         _req: &Request<'_>,
132 | |         _ino: u64,
...   |
144 | |         reply: ReplyAttr,
145 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: method is never used: `bmap`
   --> async_fuse/src/fuse_reply.rs:523:5
    |
523 |     pub async fn bmap(self, block: u64) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `error`
   --> async_fuse/src/fuse_reply.rs:582:5
    |
582 |     pub async fn error(self, err: c_int) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `size`
   --> async_fuse/src/fuse_reply.rs:599:5
    |
599 |     pub async fn size(self, size: u32) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: this function has too many arguments (8/7)
   --> fuse_ll/src/fuse/mod.rs:264:5
    |
264 | /     fn write(
265 | |         &mut self,
266 | |         _req: &Request<'_>,
267 | |         _ino: u64,
...   |
272 | |         reply: ReplyWrite,
273 | |     ) {
    | |______^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments

warning: method is never used: `data`
   --> async_fuse/src/fuse_reply.rs:606:5
    |
606 |     pub async fn data(self, bytes: Vec<u8>) -> anyhow::Result<()> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `len`
  --> async_fuse/src/fuse_request.rs:17:5
   |
17 |     pub fn len(&self) -> usize {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^

Clippy Fix 8

Fix following clippy warning

warning: method is never used: `size`
   --> fuse_ll/src/fuse/reply.rs:663:5
    |
663 |     pub fn size(self, size: u32) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: method is never used: `data`
   --> fuse_ll/src/fuse/reply.rs:668:5
    |
668 |     pub fn data(mut self, data: &[u8]) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: variant is never constructed: `NamedPipe`
  --> fuse_ll/src/fuse/mod.rs:44:5
   |
44 |     NamedPipe,
   |     ^^^^^^^^^

warning: variant is never constructed: `CharDevice`
  --> fuse_ll/src/fuse/mod.rs:46:5
   |
46 |     CharDevice,
   |     ^^^^^^^^^^

warning: variant is never constructed: `BlockDevice`
  --> fuse_ll/src/fuse/mod.rs:48:5
   |
48 |     BlockDevice,
   |     ^^^^^^^^^^^

warning: variant is never constructed: `Symlink`
  --> fuse_ll/src/fuse/mod.rs:54:5
   |
54 |     Symlink,
   |     ^^^^^^^

warning: variant is never constructed: `Socket`
  --> fuse_ll/src/fuse/mod.rs:56:5
   |
56 |     Socket,
   |     ^^^^^^

warning: field is never read: `uid`
   --> fuse_ll/src/memfs.rs:893:5
    |
893 |     uid: Uid,
    |     ^^^^^^^^

warning: field is never read: `gid`
   --> fuse_ll/src/memfs.rs:894:5
    |
894 |     gid: Gid,
    |     ^^^^^^^^

warning: field is never read: `root_path`
   --> fuse_ll/src/memfs.rs:895:5
    |
895 |     root_path: PathBuf,
    |     ^^^^^^^^^^^^^^^^^^

FUSE request buffer pool

To avoid data copy when receiving FUSE request, it needs to hold a buffer for each FUSE request.
When handling FUSE request concurrently, it needs to have multiple buffers, each for one FUSE request exclusively.
When a FUSE request is finished processing, its buffer should be return back to the buffer pool to be reused for next FUSE request.
The buffer pool size can be defined as the number of max pending requests.

Clippy Fix 7

Fix following clippy warning

warning: method is never used: `error`
   --> fuse_ll/src/fuse/reply.rs:398:5
    |
398 |     pub fn error(self, err: c_int) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> async_fuse/src/protocal.rs:120:29
    |
120 | pub const FATTR_MODE: u32 = 1 << 0;
    |                             ^^^^^^
    |
    = note: `#[warn(clippy::identity_op)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `error`
   --> fuse_ll/src/fuse/reply.rs:477:5
    |
477 |     pub fn error(self, err: c_int) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> async_fuse/src/protocal.rs:152:34
    |
152 | pub const FOPEN_DIRECT_IO: u32 = 1 << 0;
    |                                  ^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `created`
   --> fuse_ll/src/fuse/reply.rs:500:5
    |
500 |     pub fn created(self, ttl: &Duration, attr: &FileAttr, generation: u64, fh: u64, flags: u32) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> async_fuse/src/protocal.rs:194:34
    |
194 | pub const FUSE_ASYNC_READ: u32 = 1 << 0;
    |                                  ^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `locked`
   --> fuse_ll/src/fuse/reply.rs:543:5
    |
543 |     pub fn locked(self, start: u64, end: u64, typ: u32, pid: u32) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: the operation is ineffective. Consider reducing it to `1`
   --> async_fuse/src/protocal.rs:266:37
    |
266 | pub const FUSE_RELEASE_FLUSH: u32 = 1 << 0;
    |                                     ^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#identity_op

warning: method is never used: `bmap`
   --> fuse_ll/src/fuse/reply.rs:578:5
    |
578 |     pub fn bmap(self, block: u64) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: 42 warnings emitted

Fuse investigation todo

  1. Find out the difference of general io and splice io in fuse
  2. Answer the question "Can we redirect user's general io request to splice request?"
  3. Research container's io request handler.
  4. Research kernel mod to translate io request.

Implement Serializer to serialize structure to byte array.

Is your feature request related to a problem? Please describe.
There are lots of unsafe code when transforming struture to array in reply code. If we need to make it safe and efficient, a serializer is necessary.

Describe the solution you'd like
Implement the serializer for structure in FUSE friendly way.

Describe alternatives you've considered
N/A

Additional context
N/A

Metadata server on master nodes

DatenLord needs master nodes to store all the metadata about the files on each slave node.
The master nodes might use RocksDB and Raft protocal to provide metadata service.

Move memfs out of async_fuse

memfs should be out of async_fuse.

By doing so, async_fuse should define a FileSystem trait to expose FUSE operation methods to be implemented.

However, async_fuse and memfs internally use async functions, and currently Rust doesn't support defining async methods in a trait.

There is an external crate async_traits can define async methods in a trait, maybe it's a workaround for now.

Setup GitHub action

We should setup GitHub action to run some test and check while reviewing PR.
The check and test should involves cargo fmt, cargo test, cargo clippy, etc.

Add clippy check in ci

Currently there're many code style issues, we need to fix them and pass clippy. Add clippy check in github ci.

Github action should run on sub-project

CI will fail because of "error: could not find Cargo.toml in /home/runner/work/datenlord/datenlord or any parent directory"
Datenlord has two sub-projects async_fuse and fuse_ll. Cargo.toml exists in each folder instead of top folder.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.