Iterator support (TypeError: argument of type 'dict' is not iterable) about jyni HOT 10 OPEN

JyNI requires Jython, which is unfortunately not that lightweight in terms of shipping. Or does Fiji ship Jython anyway?

Yes, Fiji does ship Jython, so that dependency would not be an issue. Shipping the CPython extensions (numpy, etc.) will be more of a headache. Maybe we will be able to use conda for that.

Currently, as you might have seen in the ImageJ forum thread you referenced, there is imglyb which uses PyJNIus to start a JVM from a running python process for shared memory between python and Java. We can use this to launch ImageJ/Fiji from that Python process but I definitely see benefits in having the JVM as the main process, e.g. awt in python on OSX, so for us it is definitely worth looking into JyNI. That being said, the code changes required for imglyb to work with JyNI are very minor, so I think it is definitely worth to pursue both the Jython and CPython solutions at the same time.

from jyni.

Do you think that this is likely to happen? If so, I would prefer to wait.

That depends on us. I can push such a change to Jython if it's reasonable. It is unlikely that someone else will take hands on this dictionary stuff. The question is if we see a way to unify iterator types of PyStringMap and PyDictionary (i.e. in terms of AbstractDict). I currently don't have time to look at this stuff in detail, but if you see a good way to do it, tell me. Otherwise we will need a case distinction between these backend types in native code. (Should be doable as well.)

from jyni.

In CPython 2.7 dictobject.c there is a section starting with the comment /* Dictionary iterator types */. It defines several PyObject-types for iteration about dicts (i.e. for items, keys, values).
Since JyNI leaves dictionaries on Jython side and only wraps them by views on C-side, we should probably do it the same way with corresponding iterators. Maybe you know that Jython's idea of PyDictionary is two-split into PyStringMap and PyDictionary. These were somewhat unified under AbstractDict (an effort I undertook exactly to make things easier in JyNI). Unfortunately the unification does not include dict iterators except ValuesIter, see static class ValuesIter extends PyIterator { in org.python.core.AbstractDict. Then there is ItemsIter (class ItemsIter extends PyIterator {) in org.python.core.PyDictionary) and StringMapIter, StringMapValuesIter, KeysIter and ItemsIter in org.python.core.PyStringMap. The challenge here is to sort things out and determine in which way to back the native iterator types by the mentioned Jython ones. Won't be trivial!
(Have to leave now, but hope this can get you started a bit. Will reply to the pyjinius/imglib thoughts later...)

from jyni.

I had a look at dictobject.c. If I understand correctly, all the commented out lines define the iterator behaviour on the CPython side, and we would need to replace those lines with wrappers around the respective Jython iterators.

As this is a side project for me, I cannot promise that I will make any progress with this but I will come back to it when I have some spare time!

from jyni.

If I understand correctly, all the commented out lines define the iterator behaviour on the CPython side, and we would need to replace those lines with wrappers around the respective Jython iterators.

Yes, that is exactly right. You can use
PyObject* JyNI_PyObject_FromJythonPyObject(jobject jythonPyObject)
and
jobject JyNI_JythonPyObject_FromPyObject(PyObject* op)
to convert between Java/JNI jobject representation of Jython's Java-side PyObjects and CPython-style native PyObject* (remember to handle refcounting of PyObject* objects obtained from JyNI_PyObject_FromJythonPyObject properly).
The main issue on this front is to get the case distinction between Jython's PyDictionary and PyStringMap right. Maybe it would be better to adjust this stuff in Jython to be a bit JyNI-friendlier (but that would tie this to a future Jython 2.7.2 release).
In https://github.com/Stewori/JyNI/blob/master/JyNI-C/src/JyNI.c#L1001 there are entries required for every builtin type that must be mapped between Jython and CPython side (unlike types that exist only for Jython or only for CPython; these are handled automatically). We would have to add corresponding entries for iterator types.

from jyni.

Regarding ImageJ:
It is great to see some interest in the JyNI project.

shipping JyNI with Fiji would probably be much easier than shipping a whole CPython distribution

JyNI requires Jython, which is unfortunately not that lightweight in terms of shipping. Or does Fiji ship Jython anyway?
That said, I read through the thread at http://forum.imagej.net/t/jupyter-notebook-for-imagej/5421/16. I think the multiprocessing issue might be easy to fix, see http://bugs.jython.org/issue2600.
SciPy support requires buffer protocol support in JyNI, which is doable, but will need a serious amount of work.

from jyni.

Maybe it would be better to adjust this stuff in Jython to be a bit JyNI-friendlier (but that would tie this to a future Jython 2.7.2 release).

Do you think that this is likely to happen? If so, I would prefer to wait.

Thanks for pointing to JyNI.c:1001.

from jyni.

With https://hg.python.org/jython/rev/ddb684156587 I refined things in Jython a bit.

• dict iterators for keys, values and items now have distinct classes; this allows us to match them directly with CPython counterparts
• iterators of the same kind from PyDictionary and PyStringMap now always have a common ancestor in AbstractDict, so JyNI won't need to distinguish these cases when mapping types

from jyni.

At some point you said that some iterators were implemented to some degree, the sequence iterator in particular. There are a number of iterators in different files, the sequence iterator is the only one with code that seems to manage GC in the JyNI way, so the others may have memory leaks, but I'm not sure.

To implement more, and deal with #32, I think I need to understand GC better. I think the focus of my confusion is around the core question: where do and don't we need to increment reference counts.

What I (think I) know:

I think C variables created without malloc stick around until they are out of scope. But there is something to do with PyObjects where malloc is used so free would be needed, but I don't know the details. PyObjects also seem to store a refcnt so I think GC deals with them. When GC occurs it maps the tree of references and uses that to work out what it can free. I don't think I need to know the details of exactly how it does that (although I do know some from talks and papers on jyni.org) so long as I know/trust that things with references pointing to them won't be GC'd.

What I don't know (about PySeqIter and ref counting):

I'm not really sure what a JyObject is, or what _JyObject_New does

JyObject looks like it is a wrapper for storing a jweak, some attributes and some flags. jweak is typedef'd to jobject, this suggests it's a weak reference but I'm not sure if it is. Does it have something to do with converting between JythonPyObjects and PyObjects? I'm guessing _JyObject_new initialises one from a type and the info in builtinTypes, is there a reference count for the JyObject, if so: where is it stored and is it set to 1 when created. I suspect there isn't a reference count and that it isn't managed by GC at all(since the macro AS_JY_NO_GC is used). Presumably it is expected that when you're done with it you clean it up manually?

how are java objects created in C managed

ie how does java know there is a ref from C to the object? do we need to tell it/is there a ref count somewhere?

What is JyNIDebugOp(JY_NATIVE_FINALIZE, it, -1); doing?

Is it just there for debugging so you can know when the object is going to be finalized in debug mode, it seems to have something to do with JyReferenceMonitor, is it checking there are no outstanding references so it can throw an error if something tried to deallocate an object that is still referenced?

Sorry to give you a whole bunch of questions(and I realise it's almost a question per line of code), but understanding what is going on is the main problem at the moment.

from jyni.

These are very good questions and I'll try to answer them. However maybe not all right now.

A JyObject is an extended header over a PyObject. The additional header resides in memory really right before the PyObject, so accessing it is just an issue of memory arithmetics. The macro AS_JY contains this arithmetics. Look at https://arxiv.org/pdf/1404.6390.pdf figure 6. The JyObject would be the pointer to the left. As you already mentioned, this extended header contains the jobject and some flags and optionally an entry to a various purpose attribute list, see JyAttributes.c. Not every PyObject has a JyObject header. So an attempt to access it in invalid case can segfault or yield undefined behavior. There is some way to check this (HAS_JY?). AS_JY_WITH_GC and AS_JY_NO_GC are just shortcuts if you know for sure whether a PyGC_Head exists.

JyNIDebugOp is one of several macros and functions that feed JyReferenceMonitor with data. JyReferenceMonitor is a class that aims to keep track of memory leaks in JyNI. By default it is inactive. The monitoring must be turned on explicitly if required. See JyNIGCDemo.py and test_JyNI_gc.py for usage examples. Also see https://arxiv.org/pdf/1607.00825.pdf section 3.4.

Regarding GC handling, in JyNI there operates kind of a native part of GC that cares about modeling the native reference graph on Java side with JyGCHeads. That's in gcmodule.c. This only tangents native objects that hold native references to other native objects. E.g. PyTuple does that. E.g. PyInt does not. Now consider PyDictionary where things are less obvious. Primitive types like PyInt never hold references to anything so it is no surprise. In CPython PyDictionary is subject to native reference cycle search and in general a very relevant object type to gcmodule. In JyNI this is not the case. Here it behaves almost like PyInt because it stays fully on Java side and holds no references on native level. I think that iterators all behave like PyDictionary. But I might not be aware of every detail right now.

So much for now. We can iterate on details.

from jyni.