NumPy 1.14+ compatibility about jyni HOT 8 OPEN

I just investigated newer NumPy versions. 14.1 and 14.2 result in the same error as 14.0 above.
NumPy 14.3, 14.4, 14.5 give this new error:

('1.8.0_171', 'Oracle Corporation', ('OpenJDK 64-Bit Server VM', '25.171-b11', 'Oracle Corporation'), ('Linux', '4.4.0-128-generic', 'amd64'))
Traceback (most recent call last):
  File "/data/workspace/linux/JyNI/JyNI-Demo/src/JyNINumPyTest.py", line 73, in <module>
    import numpy as np
  File "/data/workspace/linux/numpy/1.14.5/numpy/__init__.py", line 166, in <module>
    from . import random
  File "mtrand.pyx", line 1, in init mtrand
  File "/data/workspace/linux/numpy/1.14.5/numpy/__init__.py", line 118, in <module>
    __NUMPY_SETUP__
TypeError: unhashable type: 'str'

from jyni.

So, I located the first type of failure to NumPy's source file multiarray/ctors.c.
Specifically in the function PyArray_NewFromDescr_int the line res = PyObject_Call(func, args, NULL); fails, i.e. returns zero, triggering the subsequent goto fail;. Next step is to identify why this call fails. For now I only know that it emits a bunch of Python-exceptions, where the message that we see, i.e. 'type$1' object has no attribute 'update' is only the tip of the iceberg. The full sequence of error messages is:

'numpy.ndarray' object has no attribute '_fill_value'
'numpy.ndarray' object has no attribute '_hardmask'
'numpy.ndarray' object has no attribute '_sharedmask'
'numpy.ndarray' object has no attribute '_isfield'
'numpy.ndarray' object has no attribute '_baseclass'
'type$1' object has no attribute 'update'

So far I conclude that somehow attribute (write-)access of numpy.ndarray got messed up.

from jyni.

Tracked it down a bit further:
It fails in the method MaskedArray._update_from in ma/core.py.
Specifically the line self.__dict__.update(_dict) fails. Here _dict contains a bunch of updates for attributes that shall be written to self.__dict__. It seems that for some reason self.__dict__ might not be an ordinary dictionary, which somehow causes the issue.
type(self.__dict__) results in <type 'type$1'>, which is familiar from our error message 'type$1' object has no attribute 'update'. print(self.__dict__) outputs <attribute '__dict__' of 'MaskedArray' objects>. So maybe it's an issue with attribute wrappers while the dictionary itself is fine. It seems like the key lies in understanding what type$1 actually is.

The MaskedArray class extends ndarray and I remember that I once observed Astropy to fail with JyNI because of ndarray subclassing. Maybe before version 14.0 the NumPy import worked without subclassing ndarray. Unfortunately the commit history of ma/core.py is insanely complex and it is challenging to track down the breaking commit. I thought about using git's bisect feature, but that's probably tedious because of all the rebuilds required. For now I will just continue investigation. If I can fix ndarray subclassing in JyNI, then it hopefully also improves usecases like in Astropy.

from jyni.

Note that the CPython output is as one would naively expect:
Instead of <type 'type$1'> we get <type 'dict'> and instead of <attribute '__dict__' of 'MaskedArray' objects> we get {}. So somehow JyNI, or maybe Jython already, messes up this attribute. Maybe it just misses to evaluate the attribute wrapper properly.

from jyni.

The "dict" object of type <type 'type$1'> is an instance of PyDataDescriptor, see PyDataDescriptor.java. type$1 is a generated type (as I already expected actually). PyDataDescriptor is abstract and Jython generates concrete implementations at runtime, resulting in these strange type names. Now the real question is why JyNI does not evaluate __get__. Maybe the case of extending natively defined builtin-like types is not yet properly covered, an observation that is consistent with the way how Astropy fails.
I will review the way how new-style classes are handled by JyNI. Originally the described case should have been workable, but that appears to be incomplete. Maybe this will require something like PyCPeerDerived. We'll see...

from jyni.

Quick experiment:
Inserted this code snippet into PyCPeer.__findattr_ex__ right before the return statement:

if (er != null && er.implementsDescrGet() && er.isDataDescr()){
    er = er.__get__(this, objtype);
}

Now the descriptor is evaluated, but so far this doesn't fix anything because it just evaluates to None. It results now in AttributeError: 'NoneType' object has no attribute 'update' instead of AttributeError: 'type$1' object has no attribute 'update' (Is this an improvement?).
It seems like the generated implementation of PyDataDescriptor does not know what to do with a PyCPeer...

from jyni.

The recently released NumPy 1.15.0 gives the following error on import:

/usr/lib/jvm/java-8-openjdk-amd64/bin/java:
symbol lookup error:
/data/workspace/linux/numpy/1.15.0/numpy/core/multiarray.so:
undefined symbol:
PyStructSequence_InitType

However, I suppose this is easier to fix. Adding a PyStructSequence_InitType implementation is not a big deal. However I suspect there will be subsequent issues.

from jyni.

Did you find any solution for this yet? Even a temporary hardcoded patch would do. I need to use sequitur-g2p with Jython and sadly it requires NumPy 1.14.2 or higher.

from jyni.