spark-examples / spark-scala-examples Goto Github PK

I have JDK 17 and when I try to run the simple sparksession create, I get the following error:

Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module

Please, suggest a alernative as JDK 8 soultion doesn't work for me.

regarding item: src/main/scala/com/sparkbyexamples/spark/dataframe/examples/DropColumn.scala

I am running these examples in Azure PySpark 3.3 and I noticed that df.drop('colname') does NOT drop the column from the df dataframe. It only removes it from the value returned by the current pyspark statement.

Try these three lines in pyspark:

df.drop("first_name").printSchema() #prints the schema without the first_name column, same as in your examples.

df.drop("first_name"). #run this without displaying output.
df.printSchema(). #prints the schema WITH the first_name column.

Conclusion: the df.drop('col') statement does NOT change the df dataframe.

TimeInMilli.scala is about timestamps in seconds, not milliseconds.
I ended up here because I am trying to find out how to convert a LongType value like 1582053488123 that represents time since the epoch in milliseconds to a Spark TimestampType.