nippy doesn't trust Java object passed as argument to function about clerk HOT 11 CLOSED

A small update: I thought that maybe the problem was that I was passing an external Java class instance to my function dist-plot-data, and that if I wrapped the instance and the call to its .density method in a Clojure function, it might work. i.e. I tried passing that new function that just gets the probability density of a value x to a revised dist-plot-data function. The same problem occurs, but now it's the new get-density function that sees the nippy error map where it should see the LevyDistribution object.

from clerk.

Removing .cache got rid of the problem, at least for the present. (Maybe this is a normal thing to do, but I'm pretty new with clerk--didn't know.) So I don't know whether there is a real issue here or not.

from clerk.

Sorry to keep leaving comments. I don't mean to imply that I expect a quick response. I just want to add that though I thought that being able to clear the cache might mean that this would just be an occasional annoyance, it turns out that I'm having to flush the cache and restart my repl multiple times per work session.

from clerk.

Hi @mars0i, no worries at all. I have a few things I'd like to wrap up and I'll take a closer look at your issue then.

from clerk.

I started looking at this further. Just found the nippy docs, and this page looks relevant:
http://ptaoussanis.github.io/nippy/taoensso.nippy.html

I'm not yet sure how to use the info there.

This is what the error maps look like. I should have included that before:

{:nippy/unthawable {:cause :quarantined :class-name "org.apache.commons.math3.random.Well19937c" :content #object["[B" 0x5a24f0bc "[B@5a24f0bc"] :type :serializable}}

I thought I'd pass this on so that the info is available whenever you get to the issue. Or maybe I'll figure out more later.

from clerk.

OK--I have made progress. The problem seems to be due to the fact that I'm using classes from Java libs that are not listed in taoensso.nippy/*thaw-serializable-allowlist*, because they're not in default-thaw-serializable-allowlist; these are defined in taoensso.nippy/nippy.clj. Arbitrary Java libs aren't allowed to be thawed because of the potential for a security risk--not sure I understand that, but I believe it. (I'm not sure why clerk works for a short time, and then has problems with nippy thawing. No doubt it has to do with when thawing is necessary.)

I'm using some Apache Commons libs and another Java lib (from an author I trust). Those are the ones that are appearing in the nippy error maps.

It looks like I can fix the behavior by requireing nippy, and then executing, for example:

(alter-var-root (var nippy/*thaw-serializable-allowlist*) conj "org.apache.commons.*")

That specifies that all org.apache.commons libs are safe to thaw.

I'll do more experiments, but I think that fixes the problem.

(It seemed as if doing this caused clerk to become very slow; I'll have to do more experimentation, though.)

I'll submit a PR to the nippy repo to add the Apache Commons libs to default-thaw-serializable-allowlist. The other lib I'm using is more obscure--not sure whether it will seem acceptable to add it to nippy's default thaw list.

I think this is an issue that will come up again. (It could come up with any library that uses nippy.) clerk users won't necessarily know that clerk uses nippy (or what nippy is, even), or know what to do if they start getting weird behavior due to nippy's thaw security policy. So maybe it would make sense for the clerk docs to mention that this can happen. Maybe it would even be worth adding a function to clerk that will allow a user perform the alter-var-root on *thaw-serializable-allowlist*. Not sure. That would be slightly more convenient, since it wouldn't require the user adding nippy as an explicit dependency. (I'd be happy to submit a small PR that adds such a function, if that makes sense.) There might be other issues here I don't know about, or maybe there's a much better solution.

from clerk.

Still investigating this. My solution usually works, but there are some conditions when it doesn't. Trying to pin down what they are.

from clerk.

Try evaluating those forms when things don't work:

The following places should have more pointers:

• https://ptaoussanis.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*
• https://ptaoussanis.github.io/nippy/taoensso.nippy.html#var-allow-and-record-any-serializable-class-unsafe
• taoensso/nippy#130

from clerk.

@mk Great--thanks. I'll try that.

from clerk.

I believe this has also been fixed with e5c00bb. Please reopen if it's still an issue for you.

from clerk.

OK, thanks. I haven't been using Clerk much lately, but the last time I did, it was with a recent version and I had no problems.

from clerk.