Can we make kilosort 3 deterministic? about kilosort3-docker HOT 3 CLOSED

I'm also remembering this comment from the Kilosort code about making things slower but deterministic.
https://github.com/MouseLand/Kilosort/blob/main/CUDA/mexGPUall.m#L12

In our Docker build process, this is realized here:
https://github.com/benjamin-heasly/kilosort3-docker/blob/main/kilosort3/mex-build.sh#L62

Why would deterministic be slower? I'm not sure what this is all about, but it seems to meet the metal here:
https://github.com/MouseLand/Kilosort/blob/main/CUDA/mexMPnu8.cu#L987

Something we could mess with if we want.

from kilosort3-docker.

Here's what I'll try:

• Modify our Docker image build to set both ENSURE_DETERM and ENABLE_STABLEMODE during the build of mex-gpu function mexMPnu8.
• Modify our runKilosort entrypoint to accept a value for useStableMode, to use as Kilosort's ops.useStableMode, which Kilosort passes as an argument to its mexMPnu8 mex-gpu function.
• Modify our runKilosort entrypoint to accept a value for LTseed, to use as Kilosort's ops.LTseed, which Kilosort then passes on to Matlab's rng.
• Modify our runKilosort entrypoint to accept a value gpuSeed, and to apply this value after initializing the gpu, using Matlab's gpurng('default') and gpurng(gpuSeed)

After all that I'll come back and comment on whether things seem deterministic -- either way.

from kilosort3-docker.

I sorted the same file three times, using the v0.0.32 version of this repo, which makes a best effort to be rng-consistent, as in the comment above.

It seems the sorting is still not deterministic. Informally, I saw:

• test-1: cluster count 46, good clusters 11, 39, 42
• test-2: cluster count 34, good clusters 27, 30
• test-3: cluster count 48, good clusters 11, 41, 44

FWIW, this was a 16-channel Plexon recording, about 3993s long, which we converted to Kilosort format using plx-to-kilosort v0.0.22.

From this little test I conclude that I don't know a straightforward way to make Kilosort 3 deterministic. Perhaps we'll just move on as-is until we need otherwise, or until Kilosort 4, which may be quite different, anyway!

from kilosort3-docker.