vertexclique / nuclei Goto Github PK

View Code? Open in Web Editor NEW

236.0 7.0 18.0 37.43 MB

Proactive IO & Runtime system

Home Page: https://vertexclique.github.io/nuclei/

Rust 100.00%

io proactor iouring async rust-lang runtime zero-cost-abstraction lock-free epoll kqueue

nuclei's Introduction

Nuclei: Proactive IO & Runtime system

Nuclei is a proactor-based IO system which is runtime agnostic and can work with any runtime. The proactor system's design principles are matching Boost Asio. Nuclei is not using a conventional reactor approach. It is completely asynchronous, and it's wrapping poll based IO in a proactive fashion.

Nuclei uses io_uring on Linux as the primary IO backend, secondarily you can use epoll. On MacOS, Nuclei is using kqueue. On Windows, the IOCP backend will be used.

The current io_uring implementation needs a modern Linux kernel (5.19+).

Features

Most of the recent IO_URING features are available, ZC and other 6.1+ features are in TODO.
Async TCP, UDP, Unix domain sockets and files...
The proactor system doesn't block
Scatter/Gather operations
Minimal allocation
More expressive than any other runtime
Completely asynchronous I/O system with lock free programming

Examples

Please head to examples directory to run the examples:

$ cd examples
$ cargo run --example fread-vect

Tests

$ cargo test --no-default-features --features=iouring # For iouring
$ cargo test # For others

Configurations

Evented IO backend

When the iouring feature gate is not enabled, the platforms evented backend is used. For example, on Linux, epoll would be used.

Executor

Executor is using async-global-executor. Available features are:

async-exec: Uses async-io feature of async-global-executor.
tokio

License

^{Licensed under either of Apache License, Version
2.0 or MIT license at your option.}
_{Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in this crate by you, as defined in the Apache-2.0 license, shall
be dual licensed as above, without any additional terms or conditions.}

Credits

_{^{Gif is from the documentary called "Particle Fever".}}

nuclei's People

Contributors

Stargazers

Watchers

Forkers

skade keruspe icodein hopkings2008 pinkforest drmingdrmer iacore huangweiboy2 egegunay xuanwo rohankumardubey ltdjorge yhm404 alisabunina refinism malivix shabbirhasan1 tisonkun

nuclei's Issues

Reading with io_uring produces inconsistent results

When the following code is run:

use futures::future;
use futures::io::AsyncRead;
use std::io::Write;
use std::task::Poll;

fn main() -> std::io::Result<()> {
    let (x, y) = <nuclei::Handle<std::os::unix::net::UnixStream>>::pair()?;
    futures::pin_mut!(x);

    std::thread::spawn(move || {
        std::thread::sleep(std::time::Duration::from_millis(10));
        y.into_inner().write_all(b"Hello").unwrap();
    });

    nuclei::drive(future::poll_fn(|cx| {
        let mut buf = Box::new([0; 5]);
        match x.as_mut().poll_read(cx, &mut *buf) {
            Poll::Ready(n) => {
                println!("Ready! buf = {:?}, n = {:?}", buf, n);
                Poll::Ready(n)
            }
            Poll::Pending => {
                if buf.iter().any(|&b| b != 0) {
                    println!("Pending! buf = {:?}", buf);
                }
                Poll::Pending
            }
        }
    }))?;

    Ok(())
}

One of three things happen:

It works. buf is set to [72, 101, 108, 108, 111] when Poll::Ready is returned.
The data is lost. buf is never set, and when Poll::Ready is returned buf is just zeros.
buf is set, but before Ready is returned. Once Ready is returned the data is gone, and buf is just zeros.

I have issue when I add the feature tokio

hello I have issue with nuclei when I add the feature tokio:

error[E0428]: the name `EXECUTOR` is defined multiple times
   --> /home/spielrs/.cargo/registry/src/github.com-1ecc6299db9ec823/agnostik-0.1.5/src/lib.rs:149:1
    |
143 | static EXECUTOR: Lazy<executors::BastionExecutor> = Lazy::new(|| executors::BastionExecutor);
    | --------------------------------------------------------------------------------------------- previous definition of the value `EXECUTOR` here
...
149 | static EXECUTOR: Lazy<executors::TokioExecutor> = Lazy::new(|| executors::TokioExecutor::new());
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `EXECUTOR` redefined here
    |
    = note: `EXECUTOR` must be defined only once in the value namespace of this module

error[E0277]: the trait bound `bastion::BastionExecutor: LocalAgnostikExecutor` is not satisfied
   --> /home/spielrs/.cargo/registry/src/github.com-1ecc6299db9ec823/agnostik-0.1.5/src/lib.rs:319:31
    |
319 | pub fn executor() -> &'static impl LocalAgnostikExecutor {
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `LocalAgnostikExecutor` is not implemented for `bastion::BastionExecutor`
320 |     &*EXECUTOR
    |     ---------- this returned value is of type `&bastion::BastionExecutor`
    |
    = note: the return type of a function must have a statically known size

error[E0599]: no method named `spawn_local` found for struct `once_cell::sync::Lazy<bastion::BastionExecutor>` in the current scope
   --> /home/spielrs/.cargo/registry/src/github.com-1ecc6299db9ec823/agnostik-0.1.5/src/lib.rs:300:14
    |
300 |     EXECUTOR.spawn_local(future)
    |              ^^^^^^^^^^^ method not found in `once_cell::sync::Lazy<bastion::BastionExecutor>`
    |
    = help: items from traits can only be used if the trait is implemented and in scope
note: `LocalAgnostikExecutor` defines an item `spawn_local`, perhaps you need to implement it
   --> /home/spielrs/.cargo/registry/src/github.com-1ecc6299db9ec823/agnostik-0.1.5/src/lib.rs:184:1
    |
184 | pub trait LocalAgnostikExecutor: AgnostikExecutor {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0599]: no method named `set_runtime` found for struct `once_cell::sync::Lazy<bastion::BastionExecutor>` in the current scope
   --> /home/spielrs/.cargo/registry/src/github.com-1ecc6299db9ec823/agnostik-0.1.5/src/lib.rs:308:14
    |
308 |     EXECUTOR.set_runtime(runtime)
    |              ^^^^^^^^^^^ method not found in `once_cell::sync::Lazy<bastion::BastionExecu

even if I disable the default features with default-features = false I get the same error

The version that I'm using is 0.1.1

Could add a udp server demo with using IORING_FEAT_FAST_POLL and IOSQE_BUFFER_SELECT ?

nuclei.bastion.rs isded

is linked
but is ded

Need help implement Nuclei for async-tungstenite (docs pretty sparse in some places)

Hi,

I'd like to add support for Nuclei to async-tungstenite (a crate for WebSocket clients and server), but I'm relatively new to Rust and the docs of Nuclei seem to be pretty sparse in some places.

It would be great if someone would answer a few questions I have :)

My plan, for my own crate (which will use async-tungstenite), is to optionally accept an AgnostikExecutor trait object. If no executor is passed, I will create one.

(Btw., It would be great if there was a way to access the global executer ("const EXECUTER") or a generic function that generates an executer according to the selected feature flags. Something like Agnostic::create_executer() instead of using feature flags with the executer-specific functions (as Agnostic does internally). It's kind of funny that there isn't an agnostic way to create an AgnostikExecutor :)).

My first problem is, that I don't know how to use Nuclei with my optionally passed in or by myself created executer. It seems Nuclei uses the globally created executer internally.

I didn't see any information regarding this in the docs, and none of the examples creates its own executer.

My second problem is that drive is basically undocumented and I don't know when, where, and how to use it.

Without Nuclei, I have a runtime.block_on that executes my async code.

Do I need to replace that with a call of drive, or do I need to start a separate drive future or thread? (As seen in this example)

The Bastion project looks very interesting to me, and I'm glad I found a reason to try some of its parts out :)

[io_uring] Configuration option kernel_poll_only might be enabling user-mode CQ polling

If I've understood nuclei's and io_uring's documentation correctly, I believe the last commit to master enables both kernel polling for SQ and application polling for CQ while selecting kernel_poll_only, see this line:
323b1d0#diff-cba64c21ab992eaad29fce147a08f4560a4769bc14682b8a96081a5fd02dbecdR116

kernel_poll_only should enable sqpoll and disable aggressive polling. Both kernel_poll_only and low_latency_driven set the values to the same options.

proposal: Remove tokio02 and tokio03 support

Tokio has been reaching 1.0 for some time. Maybe we can now remove the support for tokio02 and tokio03?

What is the expected behavior after all the tasks are done?

Here is a modified version of the file write example. After it is done running I sleep for 34 seconds but if we monitor the task manager one core remains pegged. Is this desired behavior or is there a way to release the pool after it has been engaged?

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
#[nuclei::main]
async fn main() -> io::Result<()> {
{
// Approximately ~75,9 MB
let dark_matter = vec![DARK_MATTER_TEXT; 100_000].join("\n");

    let path = PathBuf::from("testfile.txt");

    let fo = OpenOptions::new()
        .create(true)
        .write(true)
        .open(&path)
        .unwrap();
    let mut file = Handle::<File>::new(fo).unwrap();
    file.write_all(dark_matter.as_bytes()).await.unwrap();
    let _ = file.close().await;


    let fo = OpenOptions::new()
        .read(true)
        .open(&path)
        .unwrap();

    let mut file = Handle::<File>::new(fo).unwrap();

    let mut buf = vec![];
    assert!(file.seek(SeekFrom::Start(0)).await.is_ok());
    assert_eq!(file.read_to_end(&mut buf).await.unwrap(), dark_matter.len());
    assert_eq!(&buf[0..dark_matter.len()], dark_matter.as_bytes());
    let _ = file.close().await;

    println!("Length of file is {}", buf.len());
}
//We have no more work to do yet we have 1 core pegged at 100%.
Delay::new(Duration::from_secs(34)).await;

Ok(())

}