Ability to specify benchmarks that should run synchronously about vitest HOT 3 OPEN

It looks like multiple describe are running at the same time:

However, multiple bench inside single describe are executed sequentially, so your sample code with four bench inside single describe shouldn't interfere each other. Can you check if you have multiple describe?

Though I'm not familiar with the field of benchmarking, but I would feel everything should run sequentially by default and that probably gives better stability. Maybe we can check with what other benchmark framework does in general.

I'm playing with examples like this. Multiple bench files also run in parallel by default, but it can be made sequential by fileParallelism = false:

https://stackblitz.com/edit/vitest-dev-vitest-cdshc4?file=test.log

import { bench, describe } from "vitest"
import fs from "node:fs";

const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms))

describe("suite1", () => {
  let i = 0;
  bench("bench1", async () => {
    await fs.promises.appendFile("test.log",`file1, suite1, bench1 = ${i++}\n`)
    await sleep(500);
  }, { time: 0, iterations: 0 })

  let j = 0;
  bench("bench2", async () => {
    await fs.promises.appendFile("test.log",`file1, suite1, bench2 = ${j++}\n`)
    await sleep(500);
  }, { time: 0, iterations: 0 })
})

describe("suite2", () => {
  let i = 0;
  bench("bench1", async () => {
    await fs.promises.appendFile("test.log",`file1, suite2, bench1 = ${i++}\n`)
    await sleep(500);
  }, { time: 0, iterations: 0 })
})

from vitest.

Thank you very much. We did what you suggested and it worked fine.

We made implimented it in the vite.config.js like this. Such that we still could have our tests run in parallel

import { defineConfig } from 'vitest/config';

export default defineConfig(({mode})=>({
  test: {
    fileParallelism: mode!=="benchmark",
  },
}));

And regarding if benchmarks should run sequentially. I can see arguments for both. A good first step could be to add a small section for it in the documentation.

from vitest.