Cannot .toJs() a numpy.ndarray containing BigInt values - Error: Unrecognized format character 'O' about pyodide HOT 5 OPEN

Hmm, it seems that the numpy buffer has a format string of "O". The Buffer protocol specification says the format string is

A NULL-terminated string in struct-module-style syntax describing the contents of a single item. If this is NULL, "B" (unsigned bytes) is assumed.

The format characters listed in the struct module table include "P" for void* but not "O" for PyObject*. So it seems that numpy uses some nonstandard extensions to the python buffer protocol. @mattip @charris is there a place where this is documented? It would be helpful for consumers of the buffer protocol to have a single reference for the set of possible format strings that "accepted, well-behaving" implementations of bf_getbuffer can produce...

I see that in addition to "O" there are "Zf", "Zd", and "Zg" for complex floating point types. It's not obvious how we would handle those anyways other than making a clearer error message.

from pyodide.

There is some documentation in the array interface page, but I think this is not the whole story. As you point out, Zf, Zd and Zg do not appear there (in fact, c is character and not complex as appears there), and the np.structured arrays minilanguage is, as far as I could tell, not documented. A few examples:

>>> memoryview(np.ones([10], dtype=[('a', float), ('b', int)])).format
'T{d:a:l:b:}'
>>> memoryview(np.ones([10], dtype=[('a', float), ('b', int, [10,10])])).format
'T{d:a:(10,10)l:b:}'

from pyodide.

Ah PEP 3118 has more:
https://peps.python.org/pep-3118/

from pyodide.

Oh gross pep3118 was never fully implemented and in fact it is not entirely well specified:
python/cpython#47382

The new features of the struct-string syntax are so different that I think
we need to specify a grammar. I think it will clarify some of the open
questions.

from pyodide.

Opened a question on the status of PEP 3118:
https://discuss.python.org/t/question-pep-3118-format-strings-and-the-buffer-protocol/31264

from pyodide.