Potential dependency conflicts between pypmml-spark and py4j about pypmml-spark HOT 4 CLOSED

Solution

1. Fix your direct dependencies to be py4j==0.10.7 and pyspark ==2.4.3, to remove this conflict.
   I have checked this revision will not affect your downstream projects now.

2. Ask your upstream project pyspark to loose the version range of py4j to be >=0.10.7.

Which solution do you prefer, 1 or 2?
Please let me know your choice. Then I can submit a PR to fix it.

from pypmml-spark.

@NeolithEra thanks for your feedback. About solution 1, the dependencies are too strict, and for solution 2, I'm not clear about the fix, could you explain it detailed?

Based on the doc of pyspark, the versions 2.4.x depend on the exactly py4j==0.10.7, maybe have not tested on py4j 0.10.8, not supported, or some reasons.

Actually, it's better that PySpark takes over the dependency of py4j, that means removal py4j from dependencies of PyPMML-Spark itself. Do you have any comments about this solution?

from pypmml-spark.

@scorebot I agree with your solution. Taking over the dependency of py4j is a good choice!
I have submitted a PR following your suggestion.

from pypmml-spark.

I close this issue as the referred PR has been merged.

from pypmml-spark.