Testing the behavior of CircleCI parallelism option

In order to run CircleCI steps in parallel, you need to set a value greater than 1 to the parallelism option first.

version: 2
jobs:
  test:
    docker:
      - image: circleci/node:latest
    parallelism: 3
    ...

Then you can use the CircleCI CLI for spliting tests

circleci tests glob "__test__/*.spec.ts" | circleci tests split --split-by=timing

CircleCI advertises that by passing --split-by=timing to the circleci tests split command, your tests will be split evenly.

To split by test timings, use the --split-by flag with the timings split type. The available timings data will then be analyzed and your tests will be split across your parallel-running containers as evenly as possible leading to the fastest possible test run time

Running Tests in Parallel

There are many strategies that can be potentially used for splitting tests.

One strong candidate is the Greedy Algorithm but it's not always the most optimized one.

So I decided to run the following tests in order to investigate the CircleCI's behavior.

I prepared the following tests.

• delay 20 seconds
• delay 12 seconds
• delay 10 seconds
• delay 8 seconds
• delay 6 seconds
• delay 4 seconds

I investigated how these tests are split when the parallelism option is set as 3.

If CircleCI chooses the most optimized strategy, then the tests should complete within 20 seconds.

• container 1: delay 20 seconds = 20 secnds
• container 2: delay 12 seconds + delay 8 seconds = 20 seconds
• container 3: delay 10 seconds + delay 6 seconds + delay 4 seconds = 20 seconds

CircleCI seems to choose the Greedy Algorithm and it ended up with the following result

• container 1: delay 20 seconds = 20 secnds
• container 2: delay 12 seconds + delay 6 seconds + delay 4 seconds = 22 seconds
• container 3: delay 10 seconds + delay 8 seconds = 18 seconds