libified functions can't be pickled about libify HOT 3 CLOSED

Hi @JoshuaBWT,

Spark has issues pickling functions that use sparkcontext internally. Thanks for bringing this to my notice, I'll look into it.

In the meanwhile, would it be possible for you to post a minimal reproducible example?

from libify.

hey sorry for the delay inbetween responses, this was more specific than I thought.

It happens really when python classes are involved. Here's my example:

you have your library notebook called lib:

class Utils:
  
  def partition_processor(self, partition):
    print(list(partition))

from libify import exporter
exporter(globals())

and in your execution notebook:

import libify

utils = libify.importer(globals(), './lib')

df = spark.createDataFrame(list(range(1, 100)), 'INT')

def partition_function(partition):
  df_util = utils.Utils()
  return df_util.partition_processor(partition)

df.foreachPartition(partition_function)

maybe there is something wrong with my approach to spark's foreachpartition but the error seem very specific to libify

from libify.

Hey @JoshuaBWT, it seems this issue can be solved easily by using df_util = utils.Utils() outside the function, i.e, changing you code from

def partition_function(partition):
  df_util = utils.Utils()
  return df_util.partition_processor(partition)

to

df_util = utils.Utils()

def partition_function(partition):
  return df_util.partition_processor(partition)

The issue IMO is by calling utils.Utils() inside foreachPartition, it executes in a context which is different from the driver's main context (which is referenced by globals()), essentially causing the driver's SparkContext becoming unavailable to utils.

Here are the notebooks with a working example: Importee, Importer

And here's a screenshot showing the values getting printed (Not sure why it appears in STDERR, but unless you are really just trying to print stuff, it should be fine)

If this fix does not solve your issue, feel free to reopen.

from libify.