java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter about spark-redshift HOT 9 CLOSED

i am using spark-avro latest version but i am facing same issue. Spark-redshift is really pissing me off :(

from spark-redshift.

I suspect that this is caused by an older version of Avro being used. Over at the spark-avro repo, databricks/spark-avro#91 discusses how this problem can occur on EMR, where the older version of Avro that's provided by the EMR environment takes precedence. If you happen to be running on EMR, check out the bottom of that thread for a workaround / fix.

from spark-redshift.

If you read the spark-avro issue that I linked upthread, you'll see that the problem is due to the versions of the avro-mapred library that is provided by your runtime environment, which is independent of the spark-avro version that you're using. In order to debug this issue, you'll need to figure out where avro-mapred is coming from and see which version is being used.

If it was me, I'd use Class.forName and the steps described in https://stackoverflow.com/questions/1983839/determine-which-jar-file-a-class-is-from to figure out which JARs org.apache.avro.mapreduce.AvroKeyRecordWriter and org.apache.avro.generic.GenericData are being loaded from; once you know where the dependencies are coming from, we should be able to figure out how to ensure that a newer version of Avro gets used.

from spark-redshift.

thanks Josh :) i will try your solution

On Tue, Dec 1, 2015 at 12:54 AM, Josh Rosen [email protected]
wrote:

If you read the spark-avro issue that I linked upthread, you'll see that
the problem is due to the versions of the avro-mapred library that is
provided by your runtime environment, which is independent of the
spark-avro version that you're using. In order to debug this issue,
you'll need to figure out where avro-mapred is coming from and see which
version is being used.

If it was me, I'd use Class.forName and the steps described in
https://stackoverflow.com/questions/1983839/determine-which-jar-file-a-class-is-from
to figure out which JARs org.apache.avro.mapreduce.AvroKeyRecordWriter
and org.apache.avro.generic.GenericData are being loaded from; once you
know where the dependencies are coming from, we should be able to figure
out how to ensure that a newer version of Avro gets used.

—
Reply to this email directly or view it on GitHub
#129 (comment)
.

Regards: HAFIZ MUJADID

from spark-redshift.

I'm going to close this issue as 'not a spark-redshift issue', since I don't think that there's anything actionable here from our side. Feel free to update this thread if you'd still like some help in debugging your environment issue.

from spark-redshift.

@hafizmujadid Did you ever get this issue fixed? I'm running into the same problem, just trying to save a simple DataFrame into aws Redshift.
-Thanks!

from spark-redshift.

No, I could not get this issue fixed and left the effort to use spark_redshift

from spark-redshift.

OK, are you still writing data to Redshift from spark? If so, how are you doing it?
Thanks again!

from spark-redshift.

There are multiple options to do so.

1. you can use same approach as Spark-RedShift and save your data from in
   S3 bucket and then run copy command using redshift jdbc driver.

2. Use SaveAsHadoopDataSet as in following code

   val dbUrl = "jdbc:postgresql://" + endpoint + ":" + port + "/" +
   

   database + "?user=" + username + "&password=" + password
   val records = df.map(row => (TableRow(row.toString, columnTypes),
   null))
   val jobConf = new JobConf()
   DBConfiguration.configureDB(jobConf, "org.postgresql.Driver", dbUrl,
   username, password)
   DBOutputFormat.setOutput(jobConf, table, columns: _*)
   records.saveAsHadoopDataset(jobConf)

On Mon, May 23, 2016 at 12:46 AM, WillSidebottom [email protected]
wrote:

OK, are you still writing data to Redshift from spark? If so, how are you
doing it?
Thanks again!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#129 (comment)

Regards: HAFIZ MUJADID

from spark-redshift.