Given some code like this:

import { fork } from 'effection';

fork(function*() {
  console.log("start");
  yield someController();
  console.log("finish");
});

I'd expect this to always print finish, no matter how controller is implemented, but this is actually not the case! If controller doesn't do anything which creates a listener, then node will shut down before the controller has a chance to finish. This seems a bit surprising to me.

For example listening to an EventEmitter does not create a listener. Which is a bit surprising!

IMO, calling fork in the global context should prevent node from exiting until the fork completes, but the question is how would we implement this?

import { fork, timeout, Controller } from 'effection';
import { EventEmitter } from 'events';

let someEmitter = new EventEmitter();

function controller(): Controller {
  return (execution) => {
    let resume = () => { execution.resume("foo") };
    someEmitter.on("thing", resume);
    return () => {
      someEmitter.off("thing", resume);
    }
  };
};

let myFork = fork(function*() {
  console.log("start");
  yield controller();
  console.log("finish");
});

Yarn 1.x development is frozen and Yarn 2 doesn't seem to have the developer experience that we're looking for. Now that npm supports workspaces natively, it feels like this is the future, and so the sooner we move there, the faster it will arrive for us.

When something goes wrong, it would be nice to get a snapshot of the current running tree of effection processes that actually helps you. Right now, dumping out the bigtest tree is:

-> [1](main): running
  -> [4](): running
    -> [9](): running
      -> [12](): running
        -> [13](): running
      -> [15](): running
        -> [16](): running
      -> [18](): running
        -> [19](): running
      -> [21](): running
        -> [22](): running
      -> [85](): running
    -> [29](): running
      -> [32](commandServerErrorListener): running
        -> [33](): running
      -> [87](): running
    -> [36](): running
      -> [37](): running
        -> [90](): running
    -> [44](): running
      -> [47](): running
        -> [48](): running
      -> [50](): running
        -> [51](): running
      -> [181](): running
    -> [54](): running
      -> [57](): running
        -> [58](): running
      -> [60](): running
        -> [61](): running
      -> [63](): running
        -> [64](): running
      -> [179](): running
    -> [67](): running
      -> [70](): running
        -> [95](): running
      -> [73](): running
        -> [74](): running
      -> [76](): running
        -> [77](): running
      -> [79](): running
        -> [80](): running
      -> [141](): running
    -> [99](): running
      -> [102](): running
        -> [103](): running
      -> [105](): running
        -> [106](): running
      -> [109](): running
        -> [182](): running
      -> [184](): running
        -> [185](): running
      -> [186](): running
    -> [113](): running
      -> [144](): running
    -> [189](): running
  -> [188](): running

Which is not the most helpful. The problem is that most of the processes do not have identifiable information about them. ideally, the tree would look like:

-> [1](main): running
  -> [4](Orchestrator): running
    -> [9](ProxyServer<port: 24001>): running
      -> [12](proxy-events): running
        -> [13](): running
      -> [15](http-events): running
        -> [16](): running
      -> [18](): running
        -> [19](): running
      -> [21](): running
        -> [22](): running
      -> [85](): running
    -> [29](ComandServer<port: 24002>): running
      -> [32](commandServerErrorListener): running
....

Here at a glance, we can tell what is going on, and if we can easily identify nodes at a glance, then we can start generating all kinds of helpful diagnostic data around nodes and their lifecycles.

We drive state changes in our effection apps with generators because they are so much clearer as to what's going on, but inside effection itself we use timed callback methods to assemble what is going on. It got me thinking that even if we don't use a unified mechanism, it would be much, much clearer if we could express the context procession as a generator (or maybe an async?) function:

export function* lifecycle() {
  try {
    this.state = 'pending';
    let result = yield this.operation;
    this.state = 'waiting'
    yield awaitRequiredChildren();
    this.state = 'completed';
    return result;
  } catch (error) {
    this.state = 'errored';
    this.error = error;
    return error;
  } finally {
    yield haltAllChildren();
  }
}

When wondering why there were failing checks, I looked and saw that we still had the CircleCI webhook enabled. After disabling that webhook and check, I noticed we still have a couple things to do:

1. It looks like we're doing both branch and pr builds. This is good, but shouldn't we be only doing one build per platform? Ideally we would do branch builds, but with the the merge commit so that we can begin building as soon as a branch is pushed or commits are added to it. This is what CircleCI does. This might not be possible?
2. We should update our status badges to point to the github actions checks for all platforms. It's pointing to the CircleCI build which is quite out of date.

So our old issue with recognizing when a generator is a generator came up again :(

Currently we do this like this:

https://github.com/thefrontside/effection.js/blob/b8d72eefa6570f12a1767e5fcb6b29b5b1af4578/packages/effection/src/generator-function.js#L1-L5

But apparently, sometimes when regenerator-runtime is minified it will actually minify the constructor name as well. So our generator function gets not recognized as such. This basically boils down to an old problem:

function a({ resume }) { resume("foo") }
function* b() { yield "foo" }

main(function*() {
  yield a;
  yield b;
});

Here Effection has to "know" that it should call a with the controls (because it is a controll function), but it should convert b into a generator controller.

I think some of this problem will probably go away as we find new internal primitives, but there is a core issue here which is kind of interesting:

generator functions are functions which return a generator

In other words, the creation of the generator is lazy.

This is potentially quite profound. Consider the following code:

function* runServer() {
  let server = new Server();
  yield server.listen();
  yield server.join();
}

// ... somewhere else
yield spawn(runServer);

What happens when the server shuts down? Suppose we want to restart it automatically, we could create a supervisor which does this. Wouldn't it be nice if we could just do this:

function* runServer() {
  let server = new Server();
  yield server.listen();
  yield server.join();
}

// ... somewhere else
yield supervise(runServer);

We didn't have to change runServer at all! Yet, we could have a supervisor which simply re-runs the passed operation! This is amazingly powerful!

This only works though if all of our operations are thunks. Generator functions are already thunks of course. Let's look at how we'd write that previous example:

function a() { return ({ resume }) => resume("foo") }
function* b() { yield "foo" }

main(function*() {
  yield a;
  yield b;
});

There is no longer any ambiguity! The return value of a is a function and the return value of b is a generator, these are very easy to distinguish from each other!

This is also part of the design of trio, one of the libraries which inspired Effection, and it is mentioned in the structured concurrency blogpost that inspired Effection.

Just some random rambling thoughts!

One common pattern we have in effection is for one parent process to be a monitor / exception handler of its children. This is why the bigtest server main function body (approximately) looks like:

 try {
    let orchestrator = yield fork(Orchestrator());

    yield receive({ ready: "orchestrator" }, orchestrator);

    console.log("[cli] orchestrator ready!");

    yield;
  } catch (error) {
    console.error(error);
}

The idea here is that any error that happens in the orchestrator will fail the start function, which will cause it to be caught, printed out to the console, and then the process exits. However, with the latest version of effection, this is not what happens. The main execution does fail, and it will execute any finally blocks, but it does not throw the error into the try block of the parent at the point where it happens to be yielded. The new behavior is that the exception will be thrown only if the operation to which the generator function is yielded results in an error. In order to work as before, the above code would need to be re-written as follows:

try {
  yield function* start() {
    let orchestrator = yield fork(Orchestrator());
    yield receive({ ready: "orchestrator" }, orchestrator);
    console.log("[cli] orchestrator ready!");  
    yield;
  }
} catch (error) {
  console.error(error);
}

This is a different behavior that fell out of the implementation, the question is, is this the right behavior?

When we first added subscriptions, we were anticipating on mostly working with Subscribable types, and not so much with Subscription types, so all of the convenience methods like map and filter work on subscribables. After working with subscriptions a bit, I think this was a mistake. The natural thing to work off of is a subscription. Sometimes there isn't even a subscribable.

I also realized that it is possible to implement filter and map without having to make the actual functions operations. For example, filter could be implemented like this:

function filter<T, TReturn>(subscription: Subscription<T, TReturn>, predicate: (value: T) => boolean): Subscription<T, TReturn> {
  return {
    *next() {
      while(true) {
        let result = yield subscription.next();
        if(result.done) {
          return result;
        } else if(predicate(result.value)) {
          return result;
        }
      }
    }
  }
}

This take a subscription and return a new subscription. We could even have all of the free functions return a chainable interface, so we could have something like this:

let subscription = yield subscribe(someSubscribable);
yield subscription.map(...).filter(...).forEach(...);

I think this is reasonably ergonomic.

Thoughts? cc @cowboyd

There are now a lot of add-on helper functions for using effection in different platforms. When using node, there are certain things that we need, when in the browser there are others, and there are some that are in both. Thus far, we have definite need of the following npm packages

• @effection/events
  • on(),
  • monitorErrorsOn()
• @effection/node
  • main()

And in the future, we can generalize some of our current helpers into things like:

• @effection/websocket
• @effection/express

Let's put this stuff into a monorepo with transparent publishing.

The thing that we are calling Execution which is currently returned by fork (but probably won't be eventually) is called a Task by many systems which are similar to Effection, e.g. Rust's standardlib and ember-concurrency.

It's both a shorter and less awkward name, and also imo reflects the nature of what it represents better.

It would be awesome to have command prompt where you could input an operation, and then have that operation be evaluated and the result output to the console.

When refactor @bigtest/atom to use subscriptions, I noticed that if we created a subscription transform as an eager instance property, it would throw a weird exception whenever you tried to derive new subscriptions off of it.

However, if you defined the exact same subscription transform as a computed property, it worked just fine.

bad:

export class Atom<S> {
  subscriptions = new EventEmitter();
  states = Subscribable.from(on(this.subscriptions, 'state'))
    .map(([state]) => state as S);
}

good:

export class Atom<S> {
  subscriptions = new EventEmitter();

  get states() {
    return Subscribable.from(on(this.subscriptions, 'state'))
      .map(([state]) => state as S)
  }
}

I'm genuinely puzzled as to why one form works and the other does not. Here is a minimal reproduction on RunKit.

1. good (RunKit)
2. bad (RunKit)
3. both based on original minimal TypeScript reproduction (gist)

It's worth noting that the same problem happens in JavaScript, so it is not an issue with the TypeScript output.

Frontside has decided to move away from master branch verbiage in order to be more welcoming to all employees and contributors.

Sometimes you want to do asynchronous cleanup. For example, in @bigtestjs/server we would really like the parcel server halt code. Unfortunately, you cannot yield inside a generator that has already returned or thrown. We'd like the halt() to not be considered complete until some async code has run. e.g.

let bundler = new Bundler();
try {
  yield;
} finally {
  yield bundler.stop() <-- cannot be done.
}

The new mechanics of effection are more on point: It's a tree of "node thingies" each of which is either running, or finished running because it encountered an error, it was explicitly halted, or it returned a result and there are no child "node thingies" which are still running. Anything that changes the state of these "node thingies" must be in the form of an operation that runs in the context of an on-going computation.

But we got there by looking at the problem through a bunch of different paradigmatic lenses: algebraic effects, Erlang processes, structured concurrency, delimited continuations, promises, etc... It's no surprise that the resulting vocabulary used in the codebase for both private and public APIs reflects this diversity. So we have things like fail and ensure borrowed from Ruby, resume from delimited continuations, ExecutionContext from algebraic effect, spawn from Rust, and so on.

Lateral thinking is critical when developing something novel, but the resulting hodge-podge of terminology interferes with a shared understanding of what exactly Effection is and what problems it is effective to be used for.

In my opinion we should proceed with the APIs that we have now, since mechanics are going to be mechanics, but prior to a public release, we need to come up with a foundational analogy of what Effection trees are, and let the vocabulary flow from there. In other words, every method and every property should align with the foundational analogy. Also, we want to choose something that lowers the bar the most towards wider adoption.

Let's use this issue to discuss possible alternatives that we'd feel good about going 1.0 with. Here's an example (which is deliberately different than anything we've talked about before):

Foundational Analogy: Transaction

state:

• Transaction.isRunning
• Transaction.isWaiting
• Transaction.isSuccessful
• Transaction.isFailed
• Transaction.isCancelled
• Transaction.isOpen (isRunning | isWaiting)
• Transaction.isClosed (isSuccessful | isFailed | isCancelled)
• Transaction.parent: Transaction

controls:

• Transaction.complete(result)
• Transaction.fail(error)
• Transaction.cancel(reason)
• Transaction.run(operation)
• Transaction.onClose()

If a context is spawned and returned from the parent context all within the same event loop tick that the parent context started, it will be halted. Here is a reproduction:

import { spawn, main, join, timeout } from 'effection';

main(function*() {
  let a = function*() {
    yield timeout(2);
    return yield spawn(undefined);
  }
  let b = function*() {
    return yield spawn(undefined);
  }

  let contextA = yield spawn(a);
  let resultA = yield join(contextA);

  console.log("STATE of A:", resultA.state);

  let contextB = yield spawn(b);
  let resultB = yield join(contextB);

  console.log("STATE of B:", resultB.state);
});

We would expect this to print:

STATE of A: running
STATE of B: running

But in fact it prints:

STATE of A: running
STATE of B: halted

The only difference between a and b is the use of the timeout.

Replacing the timeout with some synchronous operation (e.g. function() {}) also halts the context.

We have channels which are basically an ephemeral message bus, but what we're lacking right now is a persistent queue. This would be somewhat similar to Mailbox in bigtest, but with a slimmed down and simplified API.

Right now, there are two mains. One from @effection/node and another from effection, which is confusing. I think that main is the understood term as an entry point to an entire process, so @effection/node main should stay the same since that really is its purpose.

That means we really ought to come up with a better name for it in effection proper. I'm thinking that we should call it run or enter. It should convey that you are entering an effection entry point, but not necessarily for the entire process.

Take the following case:

import { fork } from 'effection';

fork(function* outer() {
  fork(function* inner() {
    throw new Error('boom!');
  });
});

This aborts with an error as expected, but the error is not super useful, since it does not actually contain the original exception, but rather a complaint about incorrect Generator states:

TypeError: Generator is already running
    at outer.throw (<anonymous>)
    at thunk.iterator (fork.js:70:30)
    at Fork.fn [as enter] (fork.js:181:14)
    at Fork.enter (fork.js:158:23)
    at Fork.thunk (fork.js:70:12)
    at Fork.join (fork.js:152:7)
    at Fork.join [as finalize] (fork.js:196:19)
    at Fork.finalize [as thunk] (fork.js:172:12)
    at Fork.thunk [as resume] (fork.js:91:12)
    at Fork.resume (fork.js:89:12)

This is because when an exception occurs in a Fork, we catch it and then progressively call throw on the parent Forks all the way up to the root fork (provided that nobody catches it). There problem occurs here where the outer generator is actually part of the parent stack of the inner.

This is an invalid operation on a generator that is currently running, which outer still is. In other words, it has not yielded yet.

While this does not change the logic of a program in the sense that it will still fail at the right point, it nevertheless hides the root cause of the exception by replacing the actual error with the TypeError: Generator is already running

It's not obvious what the solution is, but some of the ones discussed so far are:

1. make all forks happen in the next tick of the run-loop. That way, the parent is guaranteed to continue to a yieldpoint before the exception is thrown and so parent.throw() will be a valid operation
2. detect when the parent is currently on the call stack, and re-throw the error to allow it to percolate to the parent after child finalization.
3. collect forked children on a queue which is then resumed in-order when the parent reaches the next yieldpoint.

We need to research these possible approaches, as well as identify others before coming to a conclusion.

We most often want to get the entire tree of a fork when debugging so that we can print it out from top to bottom.

Currently channels are defined as a single class which has both sending and receiving ability. We could view the current channel type as follows:

interface Publishable<T> {
  send(message: T): void;
}
type Channel<T> = Publishable<T> & Subscribable<T>

But what we'd really want is something like this:

function channel<T>(): [Publishable<T>, Subscribable<T>] {
  // ...
}

So that we can split up the sending and receiving end. This is useful because we often want to hand out one "end" of the channel to someone else, yet retain control of the other end of the channel. If we hand out both ends of the channel, someone else might end up e.g. sending messages to the channel, which could break internal invariants.

This also neatly opens up the possibility of duplex channels:

function duplexChannel<Tx, Rx>(): [Publishable<Tx> & Subscribable<Rx>, Publishable<Rx> & Subscribable<Tx>] {
  // ...
}

Handing out one end of a duplex channel could work similarly and would be very useful e.g. for our connection server.

It could also be an option to make Publishable a simple function interface, but this makes the duplex case a bit more challenging.

interface Publishable<T> {
  (message: T): void;
}

An open question is a good convention as to the naming of the variables that we'd assign:

let [???, ???] = channel();

@effection/node main() is designed to implement the main method of a node process and yet we don't actually do the work of coupling to process.argv which has to be done manually, but in fact this is something that you invariably need to do.

Proposal, instead of

function main(operation: Operation<T>): Context<T>;

should be:

function main((argv: string[]) => Operation<T>): Context<T>;

That way

import { main } from '@effection/node';

main(function*() {
  yargs(process.argv);
});

becomes

import { main } from '@effection/node';

main(function*(argv) {
  yargs(argv);
});

It's a bug to have an error in a teardown callback, but effection currently does not catch this, which ends up blowing up the entire teardown process leaving the tree in an inconsistent state.

We should catch errors like this and write a critical warning to the console.

One design choice that I made in mini-effection was renaming Context to Task. This aligns with Rust's async/await, with the CLR task class, and with ember-concurrency (somewhat).

I personally don't find Context to be a very intuitive name for the concept that it represents.

Maybe contrasting this with the usage of Context in GraphQL makes sense, here a context object is an object that represents some type of environment, a bundle of data that the query can access. This feels in line with what I would expect a context to be. But currently a context represents a sequence of asynchronous operations (e.g. it can be halted).

In other words, I would expect Context to define the set of memory/storage associated with a sequence of operations, not the sequence itself.

I think ExecutionContext is somewhat better, but it's both wordy and awkward, and still suffers a little bit from the same issues. Task is short and snappy, yet fits well for the intended purpose.

The difference between spawn and fork is a subtle one, and I'm dreading having to write the tutorials that explains the why of spawn vs fork. It's difficult to explain because fork does a lot of implicit magic.

The only difference between

function * withFork() {
  let child = fork(function*() {
    yield doStuff();
  });
  return 'done'
}

and

function * withSpawn() {
  let child = spawn(function*() {
    yield doStuff();
  });
  return 'done';
}

Is that withSpawn will return 'done' to its parent immediately, where as withFork will return 'done' to its parent only after doStuff() has finished. Unless you're familiar with the ins and outs of effection, this is a very surprising result. I have to imagine that this would be a major sticking point to adoption.

It's interesting to me that withFork could be re-written using spawn() plus an explicit wait, and it's a lot clearer what's going on just by looking at the code.

function * withSpawnAndWait() {
  let child = spawn(function*() {
    yield doStuff();
  });
  yield child;
  return 'done';
}

Which makes me think: "man, fork() has some seriously non-local effects, so what kind of weight is that abstraction carrying exactly?"

Given our experience with BigTest has shown that 90% of the time what we want is actually the non-blocking spawn() perhaps we should consider deprecating fork entirely and migrate existing code to using explicit, rather than implicit wait.

From thefrontside/bigtest#627

The way that this was implemented was to catch all non MainErrors, but that felt not so great. It occurred to me that what might even be better would be to have the Promise returned from @effection/nodealways resolve on both successful exits _and_ onMainError` exits since these are still expected outcomes. Only on an unexpected error would the promise actually reject. This would have the benefit of allowing custom error handling of unexpected exits, but baked in handling of all expected exits. In other words, we would have been able to write this as:

import { CLI } from './cli';
import { main } from '@effection/node';
import { unexpectedExit } from './unexpected-exit';

main(CLI(process.argv.slice(2)))
  .catch(unexpectedExit);

From the perspective of the main() operation, it would be consider a successful exit when the operation either completes, or finishes with a MainError which indicates that the program raised the error. The only time the operation would fail would be on an unexpected error. Other things to consider:

• Don't return a raw Context, but a Promise implementor that has a reference to the context. That way, when you install a catch handler, it erases the default catch handler, or when you install a then handler, it erases the default tach handler. That way, there is full control about what to do when the process resolves.

The Subscribable interface defines a minimal interface which a type must conform to in order to be subscribable. This means you can always use Subscribable.from(...) to get the nice chainable interface. However, while it shouldn't be mandatory, sometimes it would be nice to actually implement the full set of chainable operations on a type directly. As I'm porting channels to the new subscriptions, I found this to be the case. Instead of having to write:

let myChannel = new Channel();
yield Subscribable.from(myChannel).map((value) => value * 2).first();

It would be much nicer to be able to call map directly on the channel:

let myChannel = new Channel();
yield myChannel.map((value) => value * 2).first();

But this requires actually adding map to the Channel type. In the interest of not having there be any compability problems, and uniformity across types in the ecosystem, maybe we should provide an additional interface which extends Subscribable that types can implement, which also defines the map and filter methods and so on.

All of the actual implementations would of course simply delegate to Subscribable.from(this).map(fn) and so on.

We were talking yesterday about this idea of using something like a continuation to represent the different stages that an execution context goes through. I spent some time thinking about this today, and I started to think that maybe this is all unnecessary. Do iterators already provide us with everything we need?

Let's walk through an example of an iterator:

gen = function*() {
  try {
    yield 1;
    yield 2;
  } catch(e) {
    yield "error";
    throw e;
  } finally {
    yield "finally";
  }
}

This is the successful case:

> a = gen()
Object [Generator] {}
> a.next()
{ value: 1, done: false }
> a.next()
{ value: 2, done: false }
> a.next()
{ value: 'finally', done: false }
> a.next()
{ value: undefined, done: true }

And let's step through how this behaves in case of throwing an error:

> a = gen()
Object [Generator] {}
> a.next()
{ value: 1, done: false }
> a.next()
{ value: 2, done: false }
> a.throw(new Error("foo"))
{ value: 'error', done: false }
> a.next()
{ value: 'finally', done: false }
> a.next()
Thrown:
Error: foo
> a.next()
{ value: undefined, done: true }

And let's step through how this behaves in case of return:

> a = gen()
Object [Generator] {}
> a.next()
{ value: 1, done: false }
> a.next()
{ value: 2, done: false }
> a.return()
{ value: 'finally', done: false }
> a.next()
{ value: undefined, done: true }

In other words, iterators/generators already provide us with a way of doing cleanup and error handling.

What if halt and fail didn't just return/throw into those iterators, but instead also proceeded to continue driving those generators to conclusion by continuing to call next? This coupled with a timeout mechanism which would terminate after a certain amount of time would essentially give us async cleanup for free.

Now we only need some mechanism to invert a generator, so we can move from callbacks to generators, and suddenly we don't actually need anything else.

@cowboyd I'm interested in hearing your thoughts on this!

It's very common to use "infinite" streams that have the same lifetime as the operation that consumes them. In fact, in the wild thus far, this is more common than actually having a meaningful return value. It's a pain then to have the Subscription interface always need to specify two generic parameters, T and TReturn every time. If most of the time a subscription is infinite, then it probably makes sense to default it to undefined

Effection actually makes debugging async processes pretty nice since everything is in a strict hierarchy. However, the default debug output is not helpful. What we really want is a nice, concise way to display a Fork when debugging.

If I have a function which returns a controller, I currently have to write this:

fork(function*() { yield returnsController() });

It would be nice if there were some syntax sugar which would allow me to write:

fork(returnsController());

With the exact same behaviour as above.

Yay or nay?

While debugging bigtest server, it seemed to me that the order of destruction for effection operations that I observed in the event of an error or halt might not be ideal. For example, let's say that we have an execution tree with seven processes. These node can represent any stateful could process that contains other stateful process. For example, a websocket server that has active connections, that are reading data:

Everything is going along great, until there is an error in one of the leaf nodes. Maybe there was a read error from one of the sockets. We can call this "patient zero"

Assuming that there is no error handling in our graph, the constraints of structural concurrency guarantee that this error is going to cause every single process to either fail or halt. If we continue our socket server analogy, the read operation fails, which fails the message handler, which fails the connection handler, which fails the socket server entirely along with every sibling process along the way. The question is, what is the proper order?

As it currently stands, the error will propagate eagerly upwards on the graph, so that the destruction hooks for processes along the path of failure will be run first, and only then, will the children halt and have their destruction hooks run.

Red denotes a failed operation, while Yellow a halted operation. The number on each picture indicates the order in which the destruction happens.

However, this feels unsafe in the sense that processes lower down the tree can depend on data and other resources further up and so they should never be in a state of being alive while any of the ancestors are failed. In order to guarantee that, then the order of destruction would need to be more from the ground up.

I think that we might be (mostly) getting away with this right now because the entire destruction of a tree is 100% synchronous, so there is no opportunity for a yielded child process to "wake up" in a state where its parent was destroyed and it was not. By the time the event could fire that would cause it to awaken, it would have been destroyed. However, it is still a risk if there are synchronous APIS like event emitters, etc... that are created by parents and used downwards.

I think this could be problematic if we start to introduce the capability for asynchronous shutdown. So the really the question is, what is the proper order and for which scenarios?

Sorry for the utterly confusing title, it's the best I could come up with.

This isn't really a bug in Effection, but rather a consequence of some of the design patterns we've started to adopt. Since I ran into this a few times and it is potentially very confusing, I thought it best to actually document this somewhere so we can try to find a solution to it.

The problem occurrs if we're using the RAII-like pattern of creating some value object which has an implicitly created context associated for it, or an ensure block. For example, we might create an http server (value) with an associated ensure block which closes the server (context), like this:

function *createServer() {
  let server = new Server();
  yield suspend(ensure(() => server.close());
  return server;
}

In some other context we could then use the server like this:

function *main() {
  let server = createServer();
  yield fork(function*() {
    server.doSomething();
  });
}

The structured concurrency guarantees ensure that the server won't be closed before any fork in the main function has completed. So we have effectively bound the server to the scope if was created in, and ensured it will be de-allocated at the end of this scope.

Now let's imagine we want to create a special kind of server, and we want to re-use the createServer primitive:

function *createSpecialServer() {
  let server = yield createServer();
  doSomethingSpecial(server);
  return server;
}

Now we have a problem! The server returned from createServer is bound to the scope it was created in, which is the createSpecialServer scope, and the structured concurrency guarantees ensure that it won't close until that scope is complete.

BUT at the end of the scope we are RETURNING the server, and effectively passing it UP to the parent. But we just noted that the server will close at the end of the scope!

function *createSpecialServer() {
  let server = yield createServer();
  doSomethingSpecial(server);
  return server; // close will run here!!!
}

So what we return from createSpecialServer is a closed server!

Okay, so we realize that we need to hoist the server up one scope, maybe we can do this:

function *createSpecialServer() {
  let server = yield suspend(createServer());
  doSomethingSpecial(server);
  return server;
}

Unfortunately this doesn't work, since suspend will return a Context and not the server, so if we use suspend we can't actually get the value!

So fundamentally we have a problem in that this pattern simply does not compose, and when it doesn't compose it fails in a surprising, and (as should be evident by this lengthy explanation) pretty difficult to understand way.

As I said this isn't a bug in Effection per se, but it is something we need to think about as we're trying to find recommended practices in how to use Effection.

Context

Effection could use a logo and Effection's Discord channel gets buried very quickly with other discussion so I've decided to post a Github issue for it instead.

Approach

Disclaimer: Not a design person but we must do what we must do in these trying times... and also it's sort of fun.

Fundamentals

• Use Frontside brand colors.
• Incorporate six edges like the Frontside, BigTest, and Microstates.
• Keep it abstract, compact, and cute; should be simple enough doodle easily.
• Effection is "affection" ❤️. End of story. Non-negotiable.

Explored Options

1. Nodes with arrows
   

• Not compact enough without making it look like bowling ball finger holes.
• There are a LOT of these node-type logos out there and so is not unique.

2. Stacked with arrows
   

• Just like the node-type logos, the 3-stack logos are very common too and is often used for data-related companies.
• Changing the stacks to hearts and adding arrows on top of that will result in something too complex for a logo.

Proposal

Follows all the fundamentals listed above: easy to doodle, six edges, heart-shape:

I've tried adding different borders and shadows but given the navy outline the shadows didn't have the effection effect we wanted and the borders just took away from the simplicity of the design without adding value (in my opinion):

I've also tried different size variations. I find the 120 to be the right amount of bubbly:

And variation of different edges:

And here's what the logo would look like with the rest of the Frontside brands:

Over the last 6 weeks, we've had the opportunity to work with Subscriptions a lot and while it has on the whole been positive, it has also made clear that there are still a bunch of rough edges.

I've observed the following:

1. Transforming subscriptions on both sides of the yield is desirable. There are cases where it just makes sense to have both:

let subscription: Subcription = yield Subscribable.from(source).map(x).filter(y)
let next = yield subscription.next();

let subscription: ChainableSubscription = yield subscribe(source);
let next = yield subscription.map(x).filter(y).next();

2. There are too many damn interfaces, classes and functions in the API, and it's confusing to explain which is which. Subscribable, Subscription, subscribe(), ChainableSubscription, Chain. If you want to move seemlessly from left-side chaining, to right-side chaining, you have to completely change all the names and imports that you're using.

What would be really awesome is to just have just two interfaces SubscriptionTransformer and Subscription (which extends SubscriptionTransformer), and a single function subscribe which can be used on both the left and right sides of the yield.

Thus, you could say:

let subscription: Subscription = yield subscribe(source).map(X);
yield subscription.filter(Y).next();

In practice, folks would really only need to know about the Subcription interface and the subscribe function, which would grossly simplify trying to explain how to work with them.

The type declarations could look like:

export interface SubscriptionTransformer<T,TReturn> {
  filter(predicate: (value: T) => boolean): SubscriptionTransformer<T, TReturn>;

  match(reference: DeepPartial<T>): SubscriptionTransformer<T,TReturn>;

  map<R>(mapper: (value: T) => R): SubscriptionTransformer<R, TReturn>;

  first(): Operation<T | undefined>;

  expect(): Operation<T>;

  forEach(visit: (value: T) => Operation<void>): Operation<TReturn>;
}

export interface SubscriptionIterator<T, TReturn> {
  next(): IteratorResult<T,TReturn>;
}

export type Subscription<T,TReturn> = SubscriptionTransformer<T,TReturn> & SubscriptionIterator<T,TReturn>;

On the right side, the object returned by subscribe would be a SubscriptionTransformer, and on the left, it would be a Subscription

There is one catch however, we need to be able to treat this SubscriptionTransformer on the right side of the yield as an operation which is currently not possible in effection. As it stands, subscribe() can't return anything smart. However, if we supported a ToOperation symbol to allow any object to be treated as an operation, then we could achieve it by making the subscribe function return a SubscriptionTransformer that was also an operation:

export declare function subscribe(source: SubscriptionSource): SubscriptionTransformer & Operation<Subscription>

When I was working on #87 I realized something: our terminology is off. It makes sense since this all evolved organically since effection started, but with where we are headed now, I think it would make more sense for fork and spawn to swap names. Let me explain why:

Fork has a connotation of diverging paths, which might eventually merge. See the fork-join model. So the connotation of fork is that it is the act of diverging and starting an operation in parallel, and there is a corresponding join which will eventually block and wait for the operation to complete.

If we look at the behaviour of fork in effection today (or post #87 at least), then it sort of both diverges and merges. It diverges first and then sets up the parent context to block and wait for the child to merge before exiting. The current behaviour of spawn maps more cleanly to the definition from the fork-join model.

spawn on the other hand makes sense as the name for the combined operation. Since it doesn't really have any previous connotations about it.

I know this is an incredibly annoying change to make, but if we ever do want to make this change, and if others agree with me that the current terminology is backwards, then we should make this change before Effection gains adoption.

Also, personally, I find spawn to be a nicer name, and since it will be the primitive used in the majority of cases, it makes sense to give it the nicer name ;)

We're nearing the beta release of BigTest, and once that has settled, IMO we should also start condsidering the path forward for Effection, and how we proceed with both the current version of Effection and a possible future version.

Let's recap some of the issues of the current version of Effection, and why we might want to make some fundamental changes:

1. Control functions: Currently the basic unit of an operation is the control function, which takes a context as an argument and can interact with this context directly, for example by spawning new children, halting, or attaching exit handlers. The problem with control functions is that they are not limited at all in what they can do with a context. Due to a lack of fundamental power in the abstractions offered, we've had to resort to various hacks which abuse the fact that control functions can manipulate the context directly. This is problematic, because the Context's API is a mish-mash of interfaces, some of which are meant to be internal and some which are not. Therefore it is very easy to misuse these APIs and it currently takes expert-level knowledge of the Effection internals to be able to know which APIs are safe to call under which circumstances.
2. Ambiguous operation types: This ties into (1). There is an inherit ambiguity in the operation types. There is not really any good way of distinguishing a control function from a generator function. We've employed various hacks to attempt to do this safely, but it is quite clear that this is brittle, and for example slight differences in how the code is bundled can cause mysterious, hard to debug issues.
3. Unclear whether yield is async: For all of its faults (and there are many), one thing that the promise spec does get right is that all promises are always asynchronous. Even await Promise.resolve(3) actually suspends and continues in the next iteration of the event loop. Effection currently allows yield to be synchronous as long as resume is always called synchronously. Not only does this lead to issues like #26, but it is also dangerous and confusing to users. If a yield is always synchronous, a user will have to be aware of this fact and build their code around this. The current situation encourages users to rely on the fact that yield can be synchronous, but if the called code ever does decide to suspend then this has the potential of introducing subtle bugs into the program.
4. Spawning as an operation: Starting with #57 we have treated spawning as an operation. Given (3) this leads to very problematic situations where a lot of our code in BigTest relies on the fact that yield is synchronous when spawning. It's worth pointing out that there is no inherent reason for spawn to be an operation! Even today there is a non-operation API for spawn via control functions, it is just not encouraged to be used. The same can be said for subscriptions. There is no reason for creating a subscription to be an async operation.
5. Poor integration with existing promise and callback based code: Today's Effection needs to resolve to various hacks to integrate well with promise and callback based code. The fact of the matter is that there is a large ecosystem of JavaScript libraries out there, and in BigTest we've had to do a lot of work to integrate well with them. It feels like there should be a more straightforward path of doing this.
6. Asynchronous teardown: There are many situations where halting cannot be synchronous, yet present day Effection assumes that halting is always synchronous. Asynchronous requires quite drastic changes to the execution model.
7. Resources: with resources we solved the problem presented in #86. However, while resources are in some ways really great and magical, the problem is when that magic breaks down. While returning resources works great, composing resources is very tricky and very error prone. An operation which creates two resources and wants to return both of them is very difficult to write. The problem we tried to solve with resources was that implicit scopes are not carried forward, it has become increasingly clear that the better solution to this is to not use implicit scopes in the first place. Making it explicit which scope spawn occurs in, rather than using the current scope, solves the same issue that resources were meant to solve with a lot less magic and a lot less connfusion, at the expense of being more verbose and explicit.
8. TypeScript: Current Effection is written in JavaScript, but TypeScript is great, so it would be nice to have Effection written in TypeScript.

Given all of these issues, it is pretty clear that while Effection has been great and very useful for BigTest, there are some fundamental changes which will require some very fundamental changes at least, and a complete rewrite probably.

We've been aware of these issues for a while and so have been working on a replacement for Effection in the https://github.com/jnicklas/mini-effection/ repo. In #194 we did a proof of concept for whether this version of Effection could replace the current functionality in the effection monorepo and this experiment was largely successful.

My proposal is as following:

1. Work out which issues currently stand in the way of releasing the current state of Effection as v1
2. Solve these issues and release v1
3. Freeze development of v1 and start integrating mini-effection into the repo as v2 of Effection.
4. Move to v2 of Effection in BigTest. This will require a pretty massive change, mostly to remove resources, but while this change will be a bit of work, based on the experience of #194 it should not be terribly difficult.

It's a commonly emerging pattern to be able fork off an operation that itself has several parallel forks within it. Then, at some later point, you want to wait until all the children are finished. An example of this is the piping that happens in the bigtest proxy server. There are two or three async operations copying data through the pipe, but once they're all finished only then do we want to continue.

In order to support this we need a way to yield to an execution. E.g.

yield fork(function*() {
  fork(one);
  fork(two);
  yield on(emitter, "end");
}

In other words, we want to wait until all sub children are finished.

We currently just skip lint on all of the subpackages.

Fork objects can sort of act like promises, but it seems like this:

1. Only works for top level forks (I think?)
2. Does not actually conform to Promises/A+

In particular, the then method should be able to receive two functions, the second of which is called in the error case. I'm not sure if there are other issues. Making forks into proper promises would allow us to await them (interestingly this should also make it possible to yield to forks for free, see #28).

I ran into this with our test suite in bigtest/server, where it would be convenient to be able to await forks in setup blocks.

In order to tamp down the number of interfaces that you need to import and work with in order to use subscriptions, I think it will be helpful to shuffle the naming a bit, since ChainableSubscription is a bit of a mouthful, and it's unclear what exactly a subscription is within the ecosystem. I'd like to propose renaming ChainableSubscription to Subscription since that is what is most often found on the left hand side of an assignment, and what needs to be typed most often by hand. More than that though, this is the interface that folks will use most. To make room, the razor-thin iteration API, we can rename what is currently Subscription to SubscriptionIterator:

export interface SubscriptionIterator<T,TClose> {
  next(): IteratorResult<T,TClose>;
}

When we are inside an operation, and we have received a Controls, then we have the ability to spawn a new context, this context is currently a child of the current operation, however I don't think this is ever what we want. In almost all cases, we actually want the spawned context to be a sibling of the current operation. This seems a bit a bit counterintuitive at first, but it makes sense if we look at it as suspending something into the background. If the spawned context is a child of the current operation, then it can't ever outlive it.

It's worth noting that we've done various weird workarounds in a lot of places to actually spawn a sibling instead. For example in Effection's own fork implementation and in watch in bigtest server.

Here is some example code illustrating this point:

import { Controller, monitor, main } from 'effection';


function createChild() {
  return ({ resume, spawn }) => {
    let runChild: Controller = ({ ensure }) => {
      console.log("starting child");
      ensure(() => {
        console.log("finishing child");
      });
    };
    resume(spawn(monitor(runChild)));
  }
}

main(function*() {
  console.log("starting main");

  yield createChild();

  console.log("finishing main");
});

This currently prints:

starting main
starting child
finishing child
finishing main

Since the child is immediately killed after being spawned. But we really want the child to outlive the yield point and to only be killed at the end of main.

It's interesting to note that this combination also solves our desire to do RAII-like patterns, without actually having to expose the parent. Just spawning a monitor in the parent will tie the resulting resource to the lifetime of the parent. No need to be able to even see the parent context.

When you spawn a child process in windows,

1. it does not die when you call process.kill
2. causes a separate window to open up every time

Potentially related https://stackoverflow.com/questions/32705857/cant-kill-child-process-on-windows ?

Consider using [execa] ?

This will be very helpful when building visualizations of currently running Forks.

So I was looking at some of the code in bigtest-server, and it struck me that whenever we bootstrap effection at the top level, we have to do a dance to attach a signal interrup handler and stuff. Not only is this annoying, it's also error prone, since there are some subtle details we could get wrong.

In the more comprehensive effection PR, @cowboyd added a interrubtible higher order operation(?) which basically turned any operation into one with proper signal handlers.

While this is an okay solution, maybe we can do even better. I propose that we add this to Effection itself. Obviously since it uses process, it will be specific to Node, and not work when running Effection in e.g. a browser.

If we rename the current main function to run or something similar, we can make a more specific main function which lives under a separate path. So something like:

import { main } from 'effection/node'

What do you think?

There is a subtle bug in how #155 is implemented.

It is quite common to return a resource as a subscription, like this:

class Something {
  *[SymbolSubscribable]() {
    return yield resource(...);
  }
}

The problem comes in the way that the subscribe function is implemented in @effection/subscription. Somewhat simplified it boils down to:

function* subscribe(source) {
  let inner = yield source[SymbolSubscribable]();
  return new ChainableSubscription(inner);
}

The problem here is that the resource is bound to the context inside the subscribe function, and it is not returned from this context, meaning it will expire when subscribe exits. This is of course wrong.

I'd like to replace all of the hacky ad-hod World stuff that we have in BigTest with a more streamlined interface which allows us to interact with effection more easily in mocha tests. My first thought was to attempt to make a better version of the World interface, but there are a lot of problems we run into. Basically it is challenging to distinguish how and when errors should be propagated. It struck me that effection already solves all of these problems, so why not just double down on effection? We could with pretty minimal effort build something like this:

import { describe, it, beforeEach } from '@effection/mocha'

describe('describe is just a reexport', () => {
  beforeEach(function*() {
    yield someOperation()
  });

  it('is an operation', function*(task) {
    let process = daemon(task, 'some-background-process');
    yield once('some', event);
  });
});

It will make it make it much easier to test effection based code, and get all of the halting and error handling just right.

One downside of this is that the style of testing we have in BigTest is very heavily based on using beforeEach, and this style does not mesh well with this:

import { describe, it, beforeEach } from '@effection/mocha'

describe('running a daemon in before each', () => {
  beforeEach(function*(task) {
    daemon(task, 'some-background-process');
  });

  it('will have killed off the daemon once we get here', function*(task) {
    // daemon has already been halted once we get here
  });
});

There might be some way of exposing some task which encompasses the whole test and is shared between forEach and it blocks, but I don't have a clear idea on how to access or pass this task around.

The TypeScript tests are very, very slow, and they don't need to be run while developing.

We should take them out of the normal test suite, and put them in a test-types directory, and run them with a test:types script that is executed on CI in a separate job (it should still block the merge)

It's nice when debugging to see how forks relate to each other. The proposal is to add an id field to each fork as an increasing sequence.

For good cross platform support, we should make sure our tests run on windows