Segfault possibly caused by changes to `MaxOut` in v8.1.1 about thinc HOT 11 OPEN

Hi! We've released thinc 8.1.2 today which restricts blis to <0.8.0 again instead of <0.10.0. Thanks again for your report!

from thinc.

Hi @danieldk and @njsmith, thanks for the quick responses. That's a nice idea re blis 0.7.8, we shall do that now and come back to you.

from thinc.

I've run some more CI runs here. The results are as follows:

• CI #430 to #460 is with thinc 8.1.1 and blis 0.7.8. There were no failures in 30x runs.
• CI #462 to #491 is with thinc 8.1.1 and blis 0.9.1. There were 5x failures in 30x runs.

The error is rather intermittent so not 100% conclusive but this does suggest an issue with more recent blis versions.

from thinc.

@ascillitoe Could you reproduce the crash with the following env variable and post the stdout output of the run: BLIS_ARCH_DEBUG=1? This will help us narrow down which BLIS kernel is faulty - you should something along the lines of libblis: selecting sub-configuration 'haswell'. in the log.

Sure thing, I'll have a go!

from thinc.

Thanks for the quick response! Yeah, the OOB access is in one of the haswell kernels, which confirms our suspicion.

from thinc.

As further info, looking at the traceback again:

File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\spacy\ml\staticvectors.py", line 56 in forward

@RobertSamoilescu has noticed the failure appears to originate in staticvectors.py here. We're therefore wondering if the recent change to gemm (explosion/cython-blis#72) is the cause? Please let us know if it would be better to open an issue over on cython-blis, or if we can provide more info.

from thinc.

@RobertSamoilescu has noticed the failure appears to originate in staticvectors.py here. We're therefore wondering if the recent change to gemm (explosion/cython-blis#72) is the cause?

Unlikely, using uninitialized memory erroneously wouldn't cause segfaults, only garbage output. We found a bunch of issues in BLIS 0.9 where gemm would read out-of-bound. These were fixed upstream and the fixes were added to cython-blis. Since we thought the issues in BLIS were fixed, we changed the upper pin to include cython-blis 0.9.x in Thinc 8.1.1. But it seems like there are more memory issues 😢.

Thanks for reporting this!

from thinc.

Can you check whether you still see the same issue in an environment that has thinc 0.8.1 + blis 0.7.8? That would let us confirm for sure whether the issue is coming from thinc or blis.

from thinc.

Thanks for the update @svlandeg!

from thinc.

@ascillitoe Could you reproduce the crash with the following env variable and post the stdout output of the run: BLIS_ARCH_DEBUG=1? This will help us narrow down which BLIS kernel is faulty - you should something along the lines of libblis: selecting sub-configuration 'haswell'. in the log.

from thinc.

Hi again. I've set some more CI runs going here (508 to 518).

Interesting, it looks like all the runs that fail (like this one) have:

libblis: selecting sub-configuration 'haswell'.

whereas the runs that pass (like this one) have:

libblis: Hardware has 2 FMA units; using 'skx' sub-config.
libblis: selecting sub-configuration 'skx'.

So it looks like it might be the haswell kernels?

from thinc.