Cancellation with Tokio is unique and it should be discussed. It most likely fits in with the Futures, Streams, and Sinks section.

tokio-rs/tokio#483 (comment)

There are lots of good external learning resources. Those should be listed on the website somewhere.

Tracks overall discussion regarding the contents and structure of the "Getting Started" guide section.

Guides:

• Hello world!
• Runtime model
• Futures
• Tasks
• Example: A Chat Server

Issues

• The guide contains a I/O with Tokio section. This section is currently poor. The current plan is to create a guide section dedicated to I/O. Should the getting started section have a high level I/O introduction? If so, what should be covered?

• There is no mention of how to implement a client. All examples use servers.

• There is no mention of thread-local variables being used to manage task notification.

A general introduction to sinks and streams isn't present. Streams receive a short introduction in the hello world section, but sinks are never introduced. A reader without prior knowledge of these concepts could become confused.

Hi I'm trying to learn Tokio right now and I've started to go through the documentation. Something that I found confusing in the Getting Started - Hello World - Creating the stream section. The first code block in the section is:

fn main() {
    // Parse the address of whatever server we're talking to
    let addr = "127.0.0.1:6142".parse().unwrap();
    let stream = TcpStream::connect(&addr);

    // Following snippets come here...
}

and it's shortly followed up with another code block:

let hello_world = TcpStream::connect(&addr).and_then(|stream| {
    println!("created stream");

    // Process stream here.

    Ok(())
})
.map_err(|err| {
    // All tasks must have an `Error` type of `()`. This forces error
    // handling and helps avoid silencing failures.
    //
    // In our example, we are only going to log the error to STDOUT.
    println!("connection error = {:?}", err);
});

To me, it seems like the stream variable in the first block is not actually used anywhere, is that correct? If so, I think the // Following snippets come here... comment is a bit misleading 🤔

Include a guide showing how to use TLS with Tokio.

Where in the outline should this fit in?

Debugging Tokio apps when things go wrong is notoriously difficult.

How should we teach users how to do it?

Should this section include Gotchas (#14)

There's a need for an introduction to what Tokio is beyond the short synopsis on tokio.rs. The description should be without buzz-words or jargon and it should explain what Tokio is actually for.

This section should cover the following topics:

• Blocking vs non-blocking and Async vs. non-async
• The fact that the Rust std lib only supports blocking APIs for I/O
• Short examples of what Tokio allows: multiple client HTTP requests on a single thread, timers that don't require blocking or spin locking a thread, etc.

A lot of this is covered in the runtime model section, but the user has already had to go through the hello world example before getting there.

The guides should discuss backpressure.

Where should it go and what should be said?

The "Getting started" guide is switching away from TCP servers to a TCP client example. The next section, "I/O with Tokio" should include a page on TCP servers, but currently the outline does not have one.

This issue tracks the desired outline of the Tokio guides. This will remain fairly high level, with specifics tracked in GitHub issues for each guide page. High level guide discussion can take place here.

Sections

• Getting started
• I/O with Tokio
• Futures, Streams, and Sinks
• Transports
• Tracking Time
• Cookbook
• Internals

Getting started

The "getting started" section is a quick introduction to the aspects of Tokio required to be productive. This section is currently mostly written, but requires some improvements.

Discussion

I/O with Tokio

A deep dive into working with input/output with Tokio. After reading "getting started" and this section, the reader should be able to implement a network based application by implementing Future by hand.

Discussion

Futures, Streams, and Sinks

In depth guides describing:

• What are futures, streams, and sinks?
• When should each be used?
• How are they used?
• How are they implemented?

Discussion

Transports

TODO

Tracking Time

Teaches how to incorporate time into Tokio applications. This covers Delay as well as setting timeouts and Interval streams. The clock is also mentioned, including how to mock it out.

Discussion

Cookbook

TODO

Internals

TODO

Tracks overall discussion regarding the contents and structure of the "I/O with Tokio" guide section.

Pages:

• Overview
• Reading and writing data
• poll_ API
• Implementing AsyncRead / AsyncWrite
• Datagram APIs

Overview

Status: Unassigned.

Functions as an introduction / tutorial to using I/O with Tokio.

Contents

• Comes w/ TCP, UDP, Unix
• Non-blocking
  • Backed by Mio
  • based on runtime model (previous section).
• Setup a TCP server
• Connect with TcpStream
• Send some data using io::write_all
• Read data on server, print, and close socket (note on drop).

Reading and writing data

Status: Unassigned.

An in depth guide to reading and writing data using Tokio's TCP types. This section may also briefly talk about other I/O types such as File from tokio-fs`. On this page, combinators and future based APIs are used exclusively.

Contents

• io::read_exact
  • Should an io::read combinator be added?
    • Perhaps the BytesCodec should be introduced here? That
      adds Stream + Sink which hasn't been discussed yet.
• io::lines
• Writing data w/ combinators
• split()
  • Why is split needed? When should split be used?
  • io::copy
  • Spawn read / write on other tasks

poll_ API

Status: Unassigned.

Using the poll_ based APIs for reading and writing data. This page talks about the lower level API and explains how the combinators described on the previous page work. There should be plenty of examples.

Contents

• Dig more into AsyncReady trait.

  • poll_read function.
    • Tie in with runtime model.
  • Disclaimer: AsyncRead extends std::io::Read... mistake
  • When to use poll_read vs. combinators? (quick answer then link
    to combinator vs/ manual future impl discussion).
  • Example: implement ReadExact.

• Dig into AsyncWrite trait.

  • poll_write function
  • poll_flush function.
  • shutdown function
    • In std, blocking cleanup ops are done in drop handler.
    • shutdown performs this work, like flushing.
    • Socket is safely dropped after shutdown returns Ok(Ready).
    • Sometimes it isn't possible:
      • Alternative: spawn task w/ socket to do cleanup work.

Open questions

• What should this page be titled?

Implementing AsyncRead / AsyncWrite

Status: Unassigned.

The previous page focused on using AsyncRead and AsyncWrite. This page explains how to implement these two traits. It explains when you would implement AsyncRead or AsyncWrite for a type and how to do it.

Contents

• Decorating an AsyncRead implementation.
  • Example 1) Adding metrics.
    • Track total number of bytes returned by poll_read.
  • Example 2) Implementing Peek<T: AsyncRead>.
    • Peek implements AsyncRead
    • Peek provides poll_peek function that returns the next
      bytes w/o consuming.
• Decorating an AsyncWrite implementation
  • Implement basic buffering (BufWriter<T: AsyncWrite>).
    • Be sure to call out shutdown implementation.
• Implementing AsyncRead/AsyncWrite for a T: mio::Evented.
  • PollEvented utility.

_buf APIs

Status: Unassigned.

A brief introduction and tutorial to the bytes crate. This page shows how the _buf APIs provide an easier way to read / write data. This page also explains vectored I/O and why it is important. It then ties in But and BufMut to vectored I/O, explaining how Buf and BufMut traits may be backed by multiple byte slices and AsyncRead / AsyncWrite implementations may read and write these byte slices in batches.

Contents

• read_buf / write_buf.
• Brief bytes introduction.
• vectored I/O

Open questions

• What should this page be titled?

Datagram APIs

Status: Unassigned.

A guide of Tokio's datagram API. This guide will focus mostly on UDP, but can mention other datagram APIs towards the end (Unix datagram?)

Contents

TODO: contents

Would be good to have a guide how to run whole Tokio application on a single (current) thread (cpu core like Node.js does). As by default Tokio uses all available CPU cores.

A common question with Tokio is how to perform graceful shutdown.

• How to unblock tokio::run
• How to signal to tasks to shutdown

Runtimes do not have the concept of "background" tasks by design. Any task performing logic should have some way of detecting that they need to shutdown. Usually, background tasks receive data to process on a channel. They can then shutdown when the channel is closed.

What other hard parts are there to graceful shutdown?

"Tracking time" section needs to be outlined.

I'm generally quite confused about what could be a blocking call and if I should do anything with the concerned call. I recently watched @jonhoo's open-source contributions stream where he pointed out many cases where the library authors are using blocking calls and whatnot on tokio threads, etc. His solution most of the time was to use tokio_threadpool::blocking.

The idea here is to gather a set of real-world examples of blocking behavior to let even people who don't have much async experience to more easily understand what blocking really means and what to do about it.

I think it's important because more and more (web) libraries are using tokio. Even if some people don't use tokio directly, they must really understand the nature of async to get the most out of the benefits tokio provides.

Some examples that confuse me (from a web server POV):

• Executing database queries. Since the driver is communicating with the database and it's waiting for a response, does the tokio thread block? What if the DB driver uses a threadpool that is external to tokio, do I have to do anything differently when executing the query in a web library's request handler?
• Password hashing. It's a CPU-bound action. From @jonhoo's stream I saw that the argounatica library itself uses a threadpool, what should I do when hashing a password in this case? When a password hashing library doesn't use an internal threadpool, should I be the one now that uses tokio_threadpool::blocking like @jonhoo did in his stream?
• File system access. There is an issue about this topic: #21.
• HTTP request from within a tokio thread. Let's say we are using Hyper's HTTP client. In this case we have a library that returns Futures, so we can just return a Future ourselves and just chain it with and_then?
• Validating a JSON schema. This involves deserializing a JSON string into Rust objects and then matching the user provided JSON to a predefined schema. Does this play well when run on a tokio thread? What's the deciding factor of when something should be offloaded to an external cpupool?

The theme seems to repeat itself in my case: I'm not sure if I should do something about certain calls because of feeling clueless about these topics, lacking knowledge and not knowing where to look to solve such basic? issues.

Does that make sense?

From Gitter.

Should working with the filesystem be part of "I/O with Tokio" or should it be a dedicated section? Is there multiple pages worth of content?

The doc-blitz repo should contain enough information for someone to arrive at the repo and be productive.

Contributing

If a contributor has gone through the guides and has a question that isn't answered or something is confusing, or has thoughts on how to improve the guides, an issue is opened. The issue is closed when an outline is updated to address the issue or the guides are updated directly.

To contribute more generally to the guides, start at the guide outline (#7) then dig into the issues for each guide section. Discussion can happen in the various issues.

To volunteer to write a guide, comment in the issue and I will add your name next to the page. Then, start writing. Please open a PR on the website ASAP and then I can link to it from the guide section issue. Discussion can happen on the page PR.

General discussion on a guide page can be done in the appropriate issue.

I saw in examples like
.map(||{...})
.and_then(|(A,B)|{...})
.and_then(|(A,)|{...})
.and_then(|x|{...})
...
It's hard to find what is passed to the closure.

The Future trait uses a thread-local variable to track the current task. This should be discussed in the "Getting started" guide as it is a common question.

What kinds of things should be in the cookbook?

Prior discussion

The guide contains a I/O with Tokio section. This section is currently poor. The current plan is to create a guide section dedicated to I/O. Should the getting started section have a high level I/O introduction? If so, what should be covered?

There are a large number of non-async APIs that users might want to use within a larger async flow. Having a guide on how to use blocking APIs inside of a Tokio runtime would be very useful.

I have went trough the guide and noticed that we have Hello World! example which uses tokio directly with the tasks. Would be good to see exactly the same example but implemented with futures (stream) directly so people will be able to compare it and see what tokio does on background. I also went through the Futures section and it uses a little different example which is feels like completely separate thing and just confuses me more like why would.

I believe it would be good to see different ways of implementing the same things to understand how everything is related in this case (Futures (Stream), Tokio, Tasks)

When I build my own futures, especially when they involve asynchronous I/O, I often end up using oneshot channels to notify about request completions. In particular,I very frequently use the pattern where you spawn a long-running "connection" future that reads from a

futures::sync::mpsc::Receiver<(Request, future::sync::oneshot::Sender<Response>)>

and sends responses on the oneshot channels as it handles incoming requests one at a time.

Pointing users in this direction might be good. A good example to refer them to might be tokio-zookeeper, because there's also a live-coding session where I build up that pattern here.

It goes from a very high level tutorial using combinators to discussing poll_ functions and Async.

I'm not sure what the best thing to do here is. The Tokio runtime model is significantly different than any other platform, so new users often get consumed because their assumptions are not correct.

All the examples in "Getting started" are based on servers. How to implement a client is a pretty common question. There should be some mention of this in the "Getting started" guide.

The current guide doesn't use Rust Playground anywhere. I just checked and can confirm that tokio crates are available in the playground (not sure if all of them).

I think it's important to make examples self-contained so that they can be run without creating a local project just to run the thing. It's also easier to quickly fiddle around and interact with the examples to test one's understanding.

Regarding: https://github.com/tokio-rs/doc-push/blob/master/outline/futures-streams-sinks.md#working-with-futures

There is currently a point "Implementing custom combinators", but this does not do justice to the topic of implementing futures.

Future is implemented more than just for custom combinators. Entire applications can be written by only implementing Future and other poll_ based functions.

Maybe there should be an additional page in that section dedicated to writing code (or entire apps) entirely without future combinators?

Composing multiple libraries that use Futures but don't know about each other is probably a very common scenario. For instance, if I want to use hyper and the async API of redis-rs, how do I do that without spawning two runtimes?

An explanation of mpsc channels currently exists in the Getting Started section, but #48 will remove this explanation. Where should an explanation of channels go now?

Tracks discussion regarding the contents of the "Hello world!" guide in the "Getting started" section.

• Source

Should this be a guide or should it be part of the cookbook?

Having a section on "gotchas" or trouble shooting will be important. Most issues people run into with Tokio probably come down to 3-5 different problems. Having go-to explanations of these will help.

One specific set of gotchas around Tokio is the use of impl Future and lifetimes. There are issues with the borrow checker that users who have written a fair amount of Rust may encounter for the first time when using Futures. Having a section that talks through some subtleties even if they aren't really specific to Tokio would be good.

When building a library that uses Tokio, it seems very common to implement custom Futures, Streams, and Sink types (rather than just using built-in combinators). Coming from other languages I think people might expect to just chain functions, but that doesn't always seems to work with Tokio (though I can't quite put my finger on why). It may be worth having a guide specifically on implementing custom types

There are other "documentation" that could be helpful, but may be best left to the community. It could be nice to encourage others to write about certain topics and link to them.

For example: Tokio for $OTHER_LANGUAGE Developers.

Currently the getting started example is a chat server that includes several concepts that have yet to be explained like codecs. Would it better for readers if the example only used concepts explained in the guide? What could we replace the example with?

Tracks discussion regarding the contents of the "Runtime model" guide in the "Getting started" section.

• Source

The user of TaskExecutor must ensure that when a future is submitted, that it is done within the context of a call to run.

For more details, see the module level documentation.

https://docs.rs/tokio/0.1.11/tokio/runtime/current_thread/struct.TaskExecutor.html#method.current

I'm confused by the "within the context of a call to run" part. Does it just mean that I must only call current() from code being executed by tokio?

Outline the internals section.

There are people who will read the guide who don't have a super solid grasp on fundamentals. Beyond basic knowledge of Rust programming the getting started guides already assume the following:

• TCP Sockets
• Threads and thread pools
• Some advanced Rust topics e.g. impl Trait, lifetime bounds, VecDeque, etc.

Making sure the reader is either introduced to these topics or is told where to find out more about them before continuing the Tokio guides is important.

Why should a user not use Tokio? All good explanations of libraries come with a section on when the user should not use the library.

This might require a bit of extra thought since Tokio is so general and will probably become extremely pervasive in the Rust ecosystem as time goes on, but giving the user a good idea of when Tokio is not appropriate is imporant.

• When is it ok to do things synchronously?
• Is Tokio appropriate for embedded environments?

It's confusing to implement poll, when and where to use NotReady

We need a glossary page describing terms used in the documentation and code. Each term should include a short definition as well as reference doc sections where the concepts are discussed.

What should be included?

• Transport
• Event loop
• Task
• Future
• ...

Tracks overall discussion regarding the contents and structure of the "Tracking Time" guide section.

Pages:

• Overview
• Timeouts
• Intervals
• The Clock

TODO: Fill out

Overview

Status: Unassigned.

Functions as an introduction / tutorial to working with time.

Contents

TODO: Fill out

Timeouts

Status: Unassigned.

TODO: Description

Contents

TODO: Fill out

Intervals

Status: Unassigned.

TODO: Description

Contents

TODO: Fill out

The Clock

Status: Unassigned.

TODO: Description

Contents

TODO: Fill out

Open questions

• Should Delay be part of "Overview" or should it have a dedicated section?

Tracks overall discussion regarding the contents and structure of the "Futures, Streams, and Sinks" guide section.

Pages:

• Overview
• Working with Futures
• Working with Streams
• Working with Sinks
• Putting it Together

Overview

Status: Unassigned.

Futures were introduced in the getting started section. This page is a more in-depth guide into what a Future is conceptually, why it makes sense to represent async computations using futures. Then, it expand this into what Streams and Sinks are at a high level.

Contents

• What is the difference between a future, stream, and sink?
• Disclaimer: Tokio's futures are very different than what other languages provide.

Working with Futures

Status: Unassigned.

TODO: Description

Contents

• Intro
  • Summarize the overview page
• Combinators
  • What is a combinator?
  • What kind of combinators are there?
    • Survey of the various kinds of combinators and examples of how
      to use them.
  • Shortcomings of combinators (move by value, no borrowing).
  • Must write in a fully functional style.
  • Common patterns with futures
  • Reach often for custom combinators (implementing Future).
• Implementing custom combinators
  • TODO: Good rule of thumb for when to reach for this
  • TODO: What are some good examples to work thorugh?
• Tasks (vs. Futures)
  • Quick note, tasks are "just" a future representing the
    completion of the task's computation.
  • Common problem: Not mapping a future to Item = (), Error = ()

Working with Streams

Status: Unassigned.

TODO: Description

Content

• More detailed description fo streams, when to use them.
  • Examples of types that implement stream.
• Combinators.
  • What kind of combinators are there?
    • Survey of the various kinds of combinators and examples of how
      to use them.
• Implementing custom combinators
  • TODO: What should go here?

Working with Sinks

Status: Unassigned.

TODO: Description

Contents

• More detailed description fo streams, when to use them.
  • Examples of types that implement stream.
    • mpsc::Sender.
    • BytesCodec (hint at transports).
• Combinators
  • Not as many combaintors, combinators make most sense on the
    "pull" side of things vs. push.
  • fanout, buffer
• How to use Sink?
  • send, send_all, flush
• Implementing Sink
  • Implement a basic channel (backed by a VecDeque)
  • How is Fanout implemented.

Putting it Together

Status: Unassigned.

TODO: Description

Contents

• How to structure an application using Tokio?
  • Looping

TODO: Fill this out

Open questions

• Where does the topic of cancellation fit in?
• Can the section title be better?

Tracks thoughts for the "Tokio for libraries" guide.

• Libraries should usually not depend on tokio, instead depend on only the sub crates they need.
• #16

Building an application with Tokio usually means pulling in additional libraries. This guide section will provide an introduction to common crates and show how to use them from the context of an app built with Tokio.

Crates to include:

• Hyper
• Trust DNS

What else?