fluxml / fluxmlbenchmarks.jl Goto Github PK

Motivation and description

Currently, --deps-list has format like:

DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,[email protected]"

But it does confuse me sometimes, because I use ";" to separate different types of dependencies and use "," to distinguish different versions of a single dependency.

I only run benchmarks twice in this scenario, so I think it would be better if I change the format like this:

DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;[email protected]"

Possible Implementation

Maybe I can update the format like above mentioned, or remove this option directly and add multiple dependencies support in --target & --baseline.

[Just a reminder]

enable3 = "flux;nnlib:gemm,conv"
disable3 = "nnlib:conv"
eb7 = parse_enabled_benchmarks(enable3, disable3)
@test length(eb7) == 4      # flux(2), nnlib(2)

But:

parse enabled benchmarks: Test Failed at ...
  Expression: length(eb7) == 4
   Evaluated: 3 == 4

This is because now disable use the same logic as enable and also generates "FLUXML_BENCHMARKS_NNLIB", which leads to the situation where setdiff(enable, disable) removes the key FLUXML_BENCHMARKS_NNLIB.

Motivation and description

benchmark folder is chaos and maybe make user confused about its usage.

Possible Implementation

1. Move runbenchmarks-cli.jl, runbenchmarks-pr.jl and some other scripts used in workflow into scripts folder;
2. Collect all the scripts and create a entry script.