talos's Introduction

talos

Talos is enforcing some theory from literature concerning circuit breakers in the form of typeclasses and laws.

Usage

Talos is modularised. You can twist it and pick the dependencies that fit your need. But let's go step by step.

libraryDependencies += "org.vaslabs.talos" %% "taloscore" % "2.1.0"
libraryDependencies += "org.vaslabs.talos" %% "talosakkasupport" % "2.1.0"
libraryDependencies += "org.vaslabs.talos" %% "taloskamon" % "2.1.0"

The events library provides a way to stream events on what's happening in the circuit breakers. E.g. combining with the talosakkasupport you can do:

import akka.actor.typed.{ActorSystem, ActorRef}
import akka.actor.typed.scaladsl.adapter._
import akka.pattern.CircuitBreaker
import cats.effect.IO
import talos.circuitbreakers.TalosCircuitBreaker
import talos.circuitbreakers.akka._

import scala.concurrent.duration._

def createCircuitBreaker(name: String = "testCircuitBreaker")
                      (implicit system: ActorSystem[_]): AkkaCircuitBreaker.Instance = {
    AkkaCircuitBreaker(
      name,
      CircuitBreaker(
        system.scheduler.toClassic,
        5,
        2 seconds,
        5 seconds
      )
    )
}

If you have an existing solution based on Akka circuit breaker and you can extract the circuit breaker like this.

    val akkaCB: CircuitBreaker = talosCircuitBreaker.circuitBreaker.unsafeRunSync()

Otherwise you can use the TalosCircuitBreaker typeclass directly

    val action: IO[Unit] = talosCircuitBreaker.protect(IO(println("Running inside the circuit breaker")))
    action.unsafeRunSync()

Talos also supports the CircuitBreaker from monix

Complete usage example

How to code can be found here: https://github.com/vaslabs/talos/blob/master/examples/src/main/scala/talos/examples/ExampleApp.scala

Laws

If you wish to implement your own TalosCircuitBreaker typeclasses you can test them against the laws library:

libraryDependencies += "org.vaslabs.talos" %% "taloslaws" % "2.1.0" % Test

Run the demo

Spin up the docker images provided:

cd examples
docker-compose up

You can see the kamon status here and the prometheus metrics are exposed here .

How Tos

Setup docker to work with Kamon: Look at build.sbt, find dockerCommonSettings
Setup logging: Look in the example module in the resources for the logback.xml file.

Architecture

Note: The hystrix reporter is no longer supported (last supported version 1.0.0)

talos's People

Contributors

Stargazers

Watchers

talos's Issues

Configuration for hard coded values

Create laws for TalosCircuitBreaker typeclass

Replace the akka event bus in the monix module

May also require making the hystrix reporter more generic.

Create a new UI

Maybe fork the hystrix dashboard but remove all unrelated to akka stuff.

There is an abstraction leak in Kamon module

Having to start an actor is an abstraction leak. We can find a better way

Sandbox environment for easy demonstration

2 docker containers one for the hystrix dashboard and one to run a simple app that fires requests to some circuit breakers and exposes a hystrix stream

Create talos gateway

Akka based gateway to protect microservices with bulkheading and circuit breakers

Cross compile for scala 2.11, 2.12 and 2.13

Enforce circuit breaker fallbacks with a talos law

On a command failure execute fallback

it is essential to involve the system’s stakeholders when deciding how to handle calls made when the circuit is open."
Michael T. Nygard. Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers). Pragmatic Bookshelf, 2007.

Fallbacks must be executed in a very short time - consider maximum timeout to be 100ms
One of the reasons for failure is that systems may lack necessary resources to complete calls. By not constraining the fallback we are endangering the system in the same way of not having circuit breakers.

Manage the source stream better in hystrix reporter module

There are 3 issues with it:

It's using the Ask pattern
The underlying actor overflows if nobody subscribes
There is a race condition for each message , e.g. consumers of the stream may "starve"

Setup automatic version updates

Migrate builds to github actions

Add monitoring for "thread pools"

This will be an interesting challenge as we need to monitor the dispatcher used under the hood. Somehow.
And then get the kamon stats and convert them to meaningful stats for the circuit breaker dashboard