Automatically adding a continuation argument to the user's code may lead to unexpected surprises.

//no arguments defined by user
suspend function greeting() {...}

var collection:Array<Void->Void> = [greeting]; //Error: actually `greeting` is Continuation->Void

var fn:Void->Void = Reflect.field(this, 'greeting');
fn(); //Error: `greeting` expects one argument.

(from #15 (comment))

We could probably have the coroutine state exposable as an object rather just being a bunch of closure captured local vars and then provide a way to pass a custom object when creating the coroutine. This should be enough to have serializable coroutines as long as everything in the state object is serializable.

What I'm not sure yet is whether we could provide type safety for such objects.

HaxeFoundation/haxe#8330

Think about exception proagation scenarios.

since pecan is the currently the most promising macro implementation for coroutines and its idea is pretty close to what we have in mind in this proposal, we can use it (or its fork) as a proving ground before moving the implementation into the compiler.

here I outline the current key differences between pecan and the proposed design (calling it nadakoroutines for brevity):

starting

pecan

• coroutine "function references" are actually instances of macro-generated CoroutineFactory classes
• CoroutineFactory instances have run method that receives arguments and return an ICo instance

nadakoroutines

• suspend function references are of Suspend<TFunc> type
• Suspend<TFunc> have start method that receives arguments and continuation and has Void return type

suspending

pecan

• every coroutine has "magic" methods for use within coroutine, like "suspend", "terminate", "yield", "accept", etc.
• starting the coroutine returns the coroutine instance (ICo) which can be passed around and provides methods for suspending, resuming and terminating the coroutine
• calling methods marked with @:pecan.action or @:pecan.accept will automatically pass coroutine instance (ICo) as an argument to those methods, allowing for the same control as in the previous variant

nadakoroutines

• there's no direct access to coroutine, coroutines are automatically suspended when another coroutine is called
• there's a "magic fake coroutine" method in the stdlib called suspendCoroutine that can be called to explicitly suspend the current coroutine. this method receives a function to be called immediately after the coroutine is suspended and this function receives the continuation for resuming the coroutine

resuming

pecan

• resuming is normally done using coroutine instance (ICo) methods, like tick, give, take, etc.
• @:pecan.accept methods receive a "return value" callback, calling it will automatically resume the coroutine and give the value to it

nadakoroutines

• coroutines are resumed when their continuations are invoked. normally this is done implicitly by inner coroutines, however in case of suspendCoroutine, the continuation is exposed to the function given to it and must be explicitly called

passing values

pecan

• coroutines have access to accept() and yield() "magic" methods to receive or give values to the outside world. the outside world operates through ICo which has corresponding give and take methods
• @:pecan.action methods can take any arguments
• @:pecan.accept methods can take any arguments and return the value back to coroutine with a special callback

nadakoroutines

• coroutines pass arguments and return values to each other like normal functions
• communication to the outside world is done via continuations:
  • coroutine's start method receives a continuation along with the other arguments, this continuation will be invoked when the coroutine terminates, passing the return value to the outside world
  • the magic suspendCoroutine method receives a continuation that is used to both resume the coroutine and pass the "accepted" result to it

extra features

pecan

• has built-in labels+goto support :)

This proposal implies coroutines should be executed automatically at the instantiation point.
However, depending on the semantics of a specific coroutine that might not be the desired behavior.

var g = generator(); //instantiation
g.configure(options);
for(item in g) { //execution
  trace(item);
}

The moment to start a coroutine should be chosen by the specific coroutine implementation.

https://github.com/Kotlin/kotlin-coroutines/blob/master/kotlin-coroutines-informal.md

This proposal implies that the suspending function and the coroutine function are the same.

suspend function load(url) {...}

suspend function getFirstLoaded(urlList) {
  var loaders = [];
  for(url in urlList) {
    //`load` is suspending. But `getFirstLoaded()` definitely should not be suspended here because no `await` applied.
    loaders.push(load(url));
  }
  //get the async operation, which will complete, when the first of the loaders will complete.
  var any = AsyncUtils.whenAny(loaders)
  //here is where the suspension should happen
  return any.await();
}

I think these concerns should be separated.

The reason I'm kind of worried about this is that I can see myself using it on a nodejs server to write request handlers that are run with high concurrency.

The thing I like about explicit use about a callback, promise or await is that looking at the code I can see that execution is suspended. I think it's pretty important to see these places, because it's exactly the ones where race conditions may occur in an execution model such as node's (or that of a browser). On a large enough project the implicitness may create more problems than it solves. If for some reason (e.g. a large enough team) any of the functions called from a suspend function becomes suspendable itself, the change passes silently, even though it potentially introduced a bug that only surfaces under high concurrency and therefore in all likelihood won't be caught by tests either and instead occasionally occurs in production - the kind of bug that nobody wants to have to deal with.

I can see the appeal of the implicitness and in many scenarios I think I would use it myself (e.g. in functional code or in small scripts). So I wonder if we could somehow make this an opt-in/opt-out. Perhaps something like using ImplicitSuspend;/using ExplicitSuspend; would be good, because then you can make it per module or even per folder with import.hx.

Think about how we may cancel suspend. In C# it's CancellationTokenSource with CancellationToken. And what behaviour then we get after cancellation.

Source

try {
	doSomething();
} catch(e:String) {
	catchString();
} catch(e:Int) {
	catchInt();
} catch(e:Dynamic) {
	catchDynamic();
}

Short legend for next snippet:

• __ctx__ is a context of state machine (holds current variable values and other data)
• ${id} is a unique id for current transformation

Preprocessed into:

//This line indicates that next state should be executed inside of a `try...catch` to handle exceptions; 
//and `@state @catchBegin ${id}` is a state which should be executed on exception
__ctx__.enterTry(@state @catchBegin ${id});
//this line finishes current state and advances state machine to specified state
@goto @tryBegin ${id};
//this line indicates that a new state starts here. In the end it will be replaced with `case ${id}:`
@label @tryBegin ${id}
//actual code of `try` block
doSomething();
//this line indicates that exception handling should be turned off now.
__ctx__.leaveTry();
//if we reached this line, it means there were no exceptions, and we should skip `catch` and go to the line after last `catch` block.
@goto @tryCatchEnd ${id}
//this line indicates a start of a state with `catch` blocks
@label @catchBegin ${id}
//each catch is transformed to a type checking of caught exception
//transforming `if`s is a topic for another transformation.
if(Std.is(__ctx__.exception, String)) {
	catchString();
} else if(Std.is(__ctx__.exception, Int)) {
	catchInt();
} else {
	catchDynamic();
	//if no `catch(e:Dynamic)` is defined in user code, then this block is replaced with
	//__ctx__.notCaught()
	//which rethrows exception if current context is not awaited or propagates exception to a caller context otherwise
}
//indicates a start of next state
@label @tryCatchEnd ${id}
//some cleanup if needed
__ctx__.leaveTryCatch();

haxe.CallStack.callStack() and haxe.CallStack.exceptionStack() should contain valid stack including suspend points. Maybe add a Suspend(item:StackItem) constructor to StackItem enum.
Example:

class Test {
  async function root() {
    await timerDelay(3000);
    await level1();
  }

  async function level1() {
    level2();
  }
  
  async function level2() {
    throw "Terrible error";
  }
}

Exception stack for Terrible error should be like this:

Method('Test', 'root');
Suspend(Method('Test', 'level1'));
Method('Test', 'level2');

async function loadXml(url) {
	return Xml.parse(await loadString(url));
}

To support that we introduce new keywords: async and await, a new Expr node: EAwait(e:Expr) and the isAsync flag for the Function structure.

The current proposed design requires different await wrapper functions for different types of Promises, Tasks, etc., as well as additional parenthesis, both of which is quite clumsy.
It also requires to bind parameters for direct CPS-style functions, which in turn requires an additional allocation.
It doesn't account for multiple results (from e.g. a CPS-style callback), which also requires an additional allocation plus an additional function call as a workaround.

With direct support for an async keyword or meta and explicit (extensible) compiler-support for arbitrary types, none of these would be an issue.

Some languages/runtime don't have a proper ticker infrastructure to run the await stuffs.

Would you be kind and propose a basic Ticker proposal so that every "runtime" become aware they need to fill in this infrastructure to obtain a proprr promise/async/yield setup?

Personnaly, being a metal guy, I don't really have an opinion on syntax as long as it's typed and brand futuristic enough... but the ticker implementations and the potential allocations should be in the clear so we know the overhead by reading the docs :)

Thanks!

This is something that came up when I was at Shiro Games last year:

I wonder if we could support this as well because I totally understand how debugging coroutines can be annoying.