QuPathProject('./my_project', mode='r')
should guarantee that you can't accidentally overwrite your project.

As one of the users of JPype, I was wondering if you had any feedback for the JPype project.

I looked over your interfaces and it appears that you are using JPype purely as a backend thus providing a "heavy" wrapper where the whole Java API has to be rewritten in Python to expose it to the user. This mean no Java types are exposed and everything ends up having to be converted each time. Although this is certainly an option it does mean duplicating the documentation and of course a lot of extra work.

I am trying to better establish the concept of a "light" wrapper in which the Java classes themselves can serve as Python wrappers and the Java packages are basically Python package at least as far as Python is concerned. And have been working of features to make it possible to do so.

If you could quickly comment on a few of these features to see if they would find use.

1. Use of global converters.

Rather than forcing the type of the input of a Java method to be a specific type, the use of converts allows one to hit a particular overload using duck typing off of a protocol.

For example, in this code the Type is forced to be a Python ColorType which then goes through a converter to get Java color object.

    @color.setter
    def color(self, rgb: ColorType) -> None:
        """set the path color"""
        rgb = QuPathColor.from_any(rgb).to_java_rgb()  # maybe use argb?
        self.java_object.setColor(rgb)

JPype has the same problem in that many Python arguments may have a Python Path but java needs a Java Path. Global converters (likely should be called adapters but that is another mater) fix this by introducing this code

@_jcustomizer.JConversion("java.nio.file.Path", instanceof=SupportsPath)
def _JPathConvert(jcls, obj):
    Paths = _jpype.JClass("java.nio.file.Paths")
    return Paths.get(obj.__fspath__())

Now it doesn't matter whether there is a Java or Python Path object all Java methods can take both.

This is currently implemented in JPype 1.0

2. Keywords on class customizers. Most of the use of the Python wrapper is just to change the types and switch the naming convention. It seems like this could be addressed by instead customizing the Java object.

Customizes are Python classes that are used to overlay an existing Java class and make it look and work like a Python one.

@_jcustomizer.JImplementationFor('java.util.Map')
class _JMap(object):
    """ Customizer for ``java.util.Map``

    This customizer adds the Python list and len operators to classes
    that implement the Java Map interface.
    """

    def __jclass_init__(self):
        Mapping.register(self)

    def __len__(self):
        return self.size()

    def __iter__(self):
        return self.keySet().iterator()

    def __delitem__(self, i):
        return self.remove(i)

    def __getitem__(self, ndx):
        try:
            item = self.get(ndx)
            if item is not None or self.containsKey(ndx):
                return item
        except TypeError:
            pass
        raise KeyError('%s' % ndx)

    def __setitem__(self, ndx, v):
        self.put(ndx, v)

    def items(self):
        return self.entrySet()

    def keys(self):
        return list(self.keySet())

    def __contains__(self, item):
        try:
            return self.containsKey(item)
        except TypeError:
            return False

However the custom class in this case would still obey the Java coding conventions and not the Python ones.

To make the Java class even more Python like we can add a few keywords like

• nameBinding=func - provide a renaming function so that methods from the Java class can be renamed to a Python one so (setFoo => set_foo)
• beanProperties=True - automatically add properties for "get/set" to the name binding.
• converters=dict(type, func) - provide a list of result converter that take a Java return and produce a Python result instead. Thus Java string arguments would automatically be hidden.
• adapters=dict(type,[func]) - provide a list of input adapters that are used when matching a Java type to Python one. The adapter functions have a defined protocol they accept to each type argument.

May of these elements can be reused over and over such that one would just need to define the maps once and apply them

@JImplementationFor("someclass", converters=StandardConverters, adapters=StandardAdapters, beanProperties=True, nameBinding=PythonLike)
class _SomeClass:
  pass

It may even be possible to these setting be applied to an entire tree of classes.

I am looking to implement these based on user input targeting JPype 1.2.

3. Support for __init__.py and __init__.pyi in Jars. To better make Java package appear as Python, we need to be able to attach typing information and to the Jar file itself. This allows the user to just import a support function library which has stuff like setting up the JVM etc, and then everything else would appear under the java package tree.

for example..

import pyquo
from qupath.lib.images ImageData

This is currently a WIP for JPype 1.1.

Obvious you are under no obligation to convert your work from a "heavy" wrapper to a "light" wrapper, but I just was hoping to see if you would have any input on what it will take in terms of tools to make a "light" wrapper a better option for similar libraries in the future.

from @vankhoa21991

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-1c484506e8c5> in <module>
----> 1 qp = QuPathProject('./my_qupath_project', mode="a")
      2 qp.images  # <- access images via this
c:\somewhere\github\paquo\paquo\projects.py in __init__(self, path, mode, image_provider)
    197             for f in p_dir.iterdir():
    198                 if not f.match("*.backup"):
--> 199                     raise ValueError("will only create projects in empty directories")
    200             project = Projects.createProject(File(str(p_dir)), BufferedImage)
    201 
ValueError: will only create projects in empty directories

The project was a folder with images already added but no project.qpproj file.
Likely from a previous run, where QuPathProject.save wasn't called.

There should be a better error message in case this happens.

PathClasses in QuPath are uniquely identified by their name and their ancestor classes. This is case sensitive and names may not contain ":".

We can fill a project's path_classes with predefined classes. But it would still allow a user to create a class with a typo.

• Term normalization

  • Does Term normalization mean, we should provide fuzzy matching of class names and provide functionality to rename typo classes?

• Color consistency

  • Color creation is already abstracted in paquo.colors.QuPathColor but this can be sprinkled with automagically provided colormaps etc.

(from #8 )

Generate a CI job to test against various QuPath versions.

Low Priority.

Move whatever we can to pyproject.toml.

windows pytest output is now spam free: a5b5cb6

can we use pyproject.toml instead of pytest.ini?

Originally posted by @sdvillal in #23 (comment)

"provide a nice programmatic way to change paths to images in projects" (itemize from #4 )

todo:

• implementation
• tests

Provide a way to tell a QuPathProject to move Image files to faster locations dynamically.

Hypothetical interface:

with QuPathProject("./myproject") as qp:
    qp.cache_add(path="/mnt/super_fast_ssd/", auto_use=False)  # auto_use: explicitly ask for usage
    qp.cache_rank()  # test which cache is fastest?
    qp.cache_warmup()  # will move all project images to fastest cache (otherwise lazy on image access)

    ...  # do your stuff

@sdvillal what do you think? And does this need to support more complicated things than copying in the filesystem? Like ssh, scp, sftp, rsync, etc...

(from #8 )

This might actually relate to #9

(itemized from #4 )

Update

So it seems as if deleting an image file on windows in the same session that is was added via QuPathProject.add_image causes a Permission exception on Windows.

Causes I'm investigating currently:

• it's probably just Windows
• our windows tests on the CI use qupath's bioformats imageserver extension instead of openslide
• maybe bioformats imageserver .close method is not implemented correctly

Initial issue

So windows had two failing tests (AND loads of thread error messages that might be pytest related...)
1951a56 fixed one of them.

A potential reason why the windows tests fails is, that we don't have a https://anaconda.org/sdvillal/qupath QuPath v0.2.1 pkg for windows. And we're currently testing against 0.2.0-m11...

@sdvillal it looks like you cancelled the CI when it was building the windows packages. the job failed to upload
could you rerun them? ❤️

explain howto configure paquo via dynaconf

os.environ and shlex.split
but need to check if that works on windows...

• split options and pass along
• provide better error messages when startJVM fails

We want to be internally consistent in paquo so that it's immediately clear which parts access the java world and which parts are native to the python world.

Toplevel options are all combinations of prefixing or not prefixing Java classes and Python classes.

Examples could be:

• Java classes get prefixed with "J", python wrapper classes get the original name (java: JPathClass and py: PathClass)
• Java classes keep their names, python wrapper classes get a prefix (java: PathClass and py: QuPathPathClass)
• both get prefixed (java: JPathClass, py: QuPathPathClass)

Note:
maybe "Paquo" might be a prefix for python classes that keeps better readability (PaquoPathClass vs QuPathPathClass)

I have no strong opinion. As long as we stay consistent. 😃

ssia

BAD OOTB EXPERIENCE. BLOCKING ISSUE FOR v0.1

This is easy on linux, almost impossible on macos, I have not tried on windows.

As an alternative, there are OS-dependent libraries that allow to instrument GUIs (for example, back in the day I had mild success instrumenting ImageScope using pywinauto).

We should provide consistent defaults for the colormaps. For example, for each organ type, have the same lesions always mapping to the same colors, plus having a "normal vs not" alternative.

Originally posted by @sdvillal in #9 (comment)

(NOTE: this issue is only a part of the comment above)

Let's collect a bunch of things that paquo should be able to do.
From here we can open more specific issues once we converge to a solution
This is by no means sorted by importance or anything. Please feel free to suggest more and

• create annotations in a qupath project from python
• control a running qupath instance
• create detections?
• provide a nice programmatic way to change paths to images in projects
• query image sections via annotations (return image data in python np, cv2, ...)
• ...

Hi,

very nice library, just starting to use it. One thing I had noticed is that you configure logging with logging.basicConfig in your package directly.

This causes issues in my scripts and applications, as when the root-logger is already configured after loading your library, another call to basicConfig does not work and loggers defined in other libraries potentially already inherited the incorrect settings from the root-logger, such as the logging level.

https://stackoverflow.com/questions/27016870/how-should-logging-be-used-in-a-python-package

In Python 3.8, basicConfig got a 'force' to override this behaviour, but for Python 3.7, it is an issue.

file:////share/directory/someimage.svs

returns False for QuPathProjectImageEntry.is_readable()

I have tried to import the QuPathProject in to computers, but I cannot import QuPathProject .
1st computer (window) said: "
RuntimeError: Unable to start JVM
RuntimeError: Provided JAVA_OPTS prevent the JVM from starting! ['-XX:MaxRAMPercentage=50', '-Djava.library.path=c:\Program Files\QuPath\app']
"
2nd (Linux) said:
" Value error: no valid qupath installation found"
I installed qupath 0.2.2, Is there anything I need install to use your library?

From the [python data model on repr:

...If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value...

@ap-- Should we start converting, when possible, these "<Representation>" strings to look like "Representation()" instead?

Low priority ;-)

Let's see how much of 3.8 we really need...

Just to annoy you guys for a second. Found two typos in the readme: "it's goal" --> "its goal" and "make your live easy" --> "make your life" or "make your lives" depending if you believe in reincarnation or not.
Cheers!

need to investigate if that is possible.
It might be. We could write a groovy script that watches the directory and reloads the project when qpdata files have been updated...

Image is a tiff.
Need to investigate and work out why and create a test case.

Currently the default mode is 'x'
It might be better to have it be 'r' readonly. but that requires readonly mode to be implemented first.

Need to think about it a bit more

This requires both:

• verbatim display
• converting into polygons

We're using Tiles (PathDetectionObjects) to display heatmaps for now.
See https://forum.image.sc/t/best-way-to-display-additional-information-as-overlay/40901/6 for more information

If a user just wants to play with paquo in a shell, it's possible that they add images to a project and don't save the project.

>>> qp = QuPathProject('./test_project', mode='a')
>>> qp.add_image('123.svs')
>>> exit()

But this leaves a directory ./test_project with an image data subdirectory, because add_image writes right away, but without a qpproj file...
It would be nice If this could be cleaned up automagically...

For a few specific UI things it might be nice if paquo offered a way of populating the scripts folder.
minimal implementation would just be:

# this is not implemented yet
QuPathProject.add_script(name, groovy_txt)

Happens on the Unclassified default class.

and provide some smart errors.

• check version on import
• add tests against multiple qupath versions

when try to launch

paquo quickview some.svs --annotations some.json --annotations-cmd "spyxel-search -b"

it returns

ERROR: can't specify both --annotations and --annotations-cmd

Having this functionality could be useful (for me at least) since I can load pathologist annotations with --annotations-cmd "spyxel-search -b" and XAI generated annotations with --annotations some.json

This can wait until just before we have the first paquo release.

We could generate docs via sphinx or something else.
Is there any strong preference?

And should these be hosted on gh-pages or readthedocs or ...?

So currently there's not a particularly nice way to accomplish this in qupath...

refer to this image.sc post to see two additional options.

option (1) overlays a whole image on top of the other, but the transparency settings are a few clicks away in a menu (so this might be less than ideal for supporting annotators in a dynamic way)
option (2) can only be used to show overlayed pixel data in the current view and would need to be enhanced with eventlisteners to dynamically update whats drawn in the window dependend on zoom, position, windowsize, etc...

So both require additional code to be added to qupath.

• (1) a keyboard shortcut to toggle full transparency might already do.
• (2) a lot more java work to have full functionality.

A third option might be to use the currently implemented method and draw a tile per pixel.
I'll have to check if that is still performant.

A fourth option would be to quantize the heatmap and draw polygonal shapes corresponding to each value (something like an abstract terrain view). This is currently my preferred option.

But all of that depends on the amount of high frequency information in the heatmap that needs to be displayed.

There's multiple places where metadata can be put that relates to an image.

QuPath has

1. ProjectImageEntry Metadata, accessible via paquo.image.QuPathProjectImageEntry.metadata.
2. ImageData Properties, accessible via paquo.image.QuPathProjectImageEntry.properties

I haven't managed to get (2) displayed in qupath yet. need to have another look

Need to test how Qupath handles this.

Allow to use shapely geometries to query for other annotations.

return image data in python np, cv2, ...

(itemized from #4 )

I'm investigating why some tiffs under macos don't work.
At the beginning I assumed it's files size, but it turns out that the ones in my local test set that work were all made with libvips-8.9.1. under windows. And the ones that don't work were made from a conda env on linux (libvips-8.8.3).

I'll update here when I have a test case with synthetic data.

it should be obviously clear in the first example that this handles like open().

It'd be soooo much better if there'd be support for #37 readonly.

ssia

Requires https://github.com/bayer-science-for-a-better-life/paquo/projects/1 to be finished.

QuPath's java.net.URIs created via java.io.File.toURI() omit optional double slashes for empty authority (compliant: https://tools.ietf.org/html/rfc3986#appendix-B) but pathlib.Path always adds them (resulting in file:/home/user/image.svs vs file:///home/user/image.svs).

The two URIs are identical, but because they are currently returned as str in paquo, care has to be taken to use the right comparison mechanism.

We could consider always returning urlib.parse.ParseResults so that comparison won't cause confusion for paquo users

I think I should implement something like a closed state explicitly.

with statements dont work well with a jupyter workflow

We need to update the docs with conda-devenv instructions ✨ 🍰 ✨

Need to create a testcase for adding annotations to an existing project.

Hi,

I was running into the following issue.

I have a project with an existing set of annotations. I am opening the project with paquo using a context manager, iterate through the images and annotations, and for some annotations I update the class using the 'update_path_class' method of the annotation object.

Here is what I observe:

• If I just do it this way, the changes of the annotation don't get saved in the object. I get a logger notification that saving was skipped as no changes were observed in the image.
• If i create an unrelated new annotation, the new annotation and all previous updates to the existing annotations are being saved.

Did you observe this before? looking through the code, it looks like 'is_changed' does not get trigger if only existing annotations get changed, but no new ones got added.

Thank you!

There are many incarnations for this issue, but for a start, we should:

Given a list of image ids:

• (1) Create a project cleverly modifying the image paths in a "cache hierarchy"
• (2) Fill class names with the known lesions, including term normalization and with color consistency
• (3) Add some metadata to the images (e.g. "annotation status")

This depends on us having a standard API to specify retrieval of these data.

High priority.

ssia

I do not know how well this would work with setuptools_scm, but should we try to sync with the qupath version we are targeting? e.g. since now we are developing against 0.2.1, should we have our version be 0.2.1-something as well?

we need to decide what's nicest for the user.

Best would probably be to auto discover in the default installation paths for qupath, and then verify if the qupath version is compatible.

override could be done via envvar and/or provide some pre java import interface to set the path:
(possible way this could be implemented --- it's not yet)

import paquo
paquo.cue_qupath_path("~/my_custom_qupath_installation")  # :P

from paquo.projects import QuPathProject
...