Index out of bounds error: multiprocessing/pool.py about mountainsort HOT 4 OPEN

It's possible there is a bug with p_mask_out_artifacts.py
Maybe it only happens when it is trying to mask out an artifact toward the end of the recording?
That would be this file: https://github.com/magland/ml_ephys/blob/master/ml_ephys/preprocessing/p_mask_out_artifacts.p

Perhaps you could take a look.

from mountainsort.

Thank you for your quick response! I've looked through that function and I know the the error is happening somewhere at/within mask_chunk (line 164) but I haven't been able to figure out why it's happening; I'm finding it hard to determine which chunk the code is failing on since processing is parallelized (python isn't my "native" language, so someone else might have better luck with this). I have managed to find a temporary fix for it, though; I just put the statement throwing the error in a try/except block:

The statement only throws that exception once and nothing looks very wonky in my data, so I'm using this for now. I'm not sure which variable is being used to index the chunks, so I wasn't able to tell which chunk it's failing for. Also, just FYI, I don't think this is an issue with data size; I took one of the datasets that threw this exception, split it into 3 equal parts, and ran the algorithm on each part separately; the function only failed on the latter 2 parts.

I hope this helps narrow things down!

from mountainsort.

Thanks. I think chunk_size on line 166 (of your snippet) should be replaced by t2-t1.

from mountainsort.

Mmm.. that actually led to many more exceptions, lol. But it did provide some useful information: chunk_size is always at 2,000, num_write_chunks is always 150, write_chunk_size is always at 300,000, and t2-t1 is at 300,000 except for the last chunk, which is smaller (201,166 in my case). You're right; the code is only failing for this last chunk. It seems that get_masked_indices is returning indices that are <300,000, but often larger than the size of the last chunk.

I tried changing 'write_chunk_size' (usually at 300,000 as well) to t2-t1, but that still didn't work (still throws an exception for the last block). It might have to do with num_write_chunks (set at 150), but I'm afraid I don't know how the code works well enough to fix that (I tried (t2-t1)/chunk_size instead, but that led to more errors). It seems like the function just needs an if-statement for exceptions with the last block..

Thanks for your help and feedback!

from mountainsort.