Serialization issues with Mono 5 about mobius HOT 1 OPEN

Ok, so for anyone else out there thinking their pyspark style lambda expressions are going to work you are going to have a bad day. There is a fundamental issue, namely the C# compiler generates anonymous functions as non-serializable. SparkCLR does some work to serialize fully closed lambda expressions, but any lambda's that reference outside their scope will be picked up by the C# compiler and made non-serializable.

I'm sure there are ways around this to elegantly turn any anonymous function into a method on a serializable class, but it has thus far eluded me. The bottom line is that you need to make a serializable helper class and use a method from that class directly in map, etc. without any lambda expressions that are not fully closed.

From the sharp CLR samples there is a BroadcastHelper class that can help you transfer data as a broadcast variable, and you can use this architecture to send any sort of data to the worker threads by first initializing a new object with the data you want used in the delegate:
[Serializable] internal class BroadcastHelper<T,U> { private readonly Broadcast<T> broadcastVar; internal BroadcastHelper(Broadcast<T> broadcastVar) { this.broadcastVar = broadcastVar; } internal T Execute(U i) { return broadcastVar.Value; } }

now, this can work:
string parallel = "test serialization string"; var broadcast = sc.Broadcast(parallel); Console.WriteLine(test.Map(new BroadcastHelper<string,int>(broadcast).Execute).First());

whereas
string parallel = "test serialization string"; var broadcast = sc.Broadcast(parallel); Console.WriteLine(test.Map(x=>broadcast.Value).First());
== bad day

I would love to chat with anyone out there who has the experience to make these kind of lambda expressions work out of the box without making helper classes.

from mobius.