Comments (3)
Hi @JoshuaBWT,
Spark has issues pickling functions that use sparkcontext internally. Thanks for bringing this to my notice, I'll look into it.
In the meanwhile, would it be possible for you to post a minimal reproducible example?
from libify.
hey sorry for the delay inbetween responses, this was more specific than I thought.
It happens really when python classes are involved. Here's my example:
you have your library notebook called lib
:
class Utils:
def partition_processor(self, partition):
print(list(partition))
from libify import exporter
exporter(globals())
and in your execution notebook:
import libify
utils = libify.importer(globals(), './lib')
df = spark.createDataFrame(list(range(1, 100)), 'INT')
def partition_function(partition):
df_util = utils.Utils()
return df_util.partition_processor(partition)
df.foreachPartition(partition_function)
maybe there is something wrong with my approach to spark's foreachpartition
but the error seem very specific to libify
from libify.
Hey @JoshuaBWT, it seems this issue can be solved easily by using df_util = utils.Utils()
outside the function, i.e, changing you code from
def partition_function(partition):
df_util = utils.Utils()
return df_util.partition_processor(partition)
to
df_util = utils.Utils()
def partition_function(partition):
return df_util.partition_processor(partition)
The issue IMO is by calling utils.Utils()
inside foreachPartition
, it executes in a context which is different from the driver's main context (which is referenced by globals()
), essentially causing the driver's SparkContext
becoming unavailable to utils
.
Here are the notebooks with a working example: Importee, Importer
And here's a screenshot showing the values getting printed (Not sure why it appears in STDERR, but unless you are really just trying to print stuff, it should be fine)
If this fix does not solve your issue, feel free to reopen.
from libify.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libify.