algorithmiaio / algorithmia-ratchet Goto Github PK

The current test system is functional, as you can provide a payload and set a timeout - however that's not very expressive. This ticket explores the idea of using generalized "benchmark" algorithms that can test load against a particular algorithm with more control over test details such as ramp time, etc.

For an example of what that could look like:
https://github.com/algorithmiaio/algorithmia-TestBench/blob/main/algorithms/ParallelPipelining/src/ParallelPipelining.py

The current way that we require algorithm URIs to be defined in both workflow payloads, and algorithm files themselves for pipelining, is confusing and not very ergonomic. We should investigate a better way to define algorithm names, and being more consistent between data file paths, workflow payloads, and embedded algorithm URIs

Defining a workflow today is complex, and has a lot of gotchas, such as the order of algorithms; and how data files must be stored in a data collection on production; maybe YAML might be more legible? This ticket investigates those gotchas and attempts to simplify the workflow creation process.

Currently we only support 1 language; this ticket would enable support for the Scala Algorithm development language & interface, this would include support for build.sbt, and the scala directory structure system.

Currently we only support 1 language; this ticket would enable support for the Java Algorithm development language & interface, this would include support for pom.xml, and the java directory structure system.

This task will encompass improving on the load shaping and testing capabilities, by creating a dedicated testing algorithm to replicate the behavior of various load conditions, as well as improving the ability to define tests.
Looking at https://github.com/algorithmiaio/load-tester#performance-testing-a-new-cluster-against-our-standard-benchmark
and https://github.com/algorithmiaio/algorithmia-benchmark for reference