Fully managed cloud. 30-day FREE full featured trial. Start Now
cancel
Showing results for 
Search instead for 
Did you mean: 

com.exasol#spark-connector_2.12;1.2.0: not found

achu310197
Padawan

Hi,

I've been trying connect to exasol via spark using spark-exasol connector jar. I'm using Spark 2.4 on my emr cluster.

 

Below is the script(spark_exasol_con_app.py) to connect:

from pyspark.sql import *

sparkSession = SparkSession.builder.getOrCreate()
df = sparkSession\
.read\
.format("exasol")\
.option("host", "<host_name>")\
.option("port", "<port>")
.option("username", "<user_name>")\
.option("password", "<password>")\
.option("query", "SELECT * from <db_name>.<table_name> limit 10")\
.load()
df.show()

 

Below is the command I'm trying to run:

spark-submit --master yarn --jars s3://<bucket>/test/spark-connector_2.12-1.2.0-spark-2.4.8-assembly.jar --packages com.exasol:spark-connector_2.12:1.2.0 --class com.myorg.SparkExasolConnectorApp s3://<bucket>/test/spark_exasol_con_app.py

 

Spark submitting the script using the command written above I'm getting below error:

com.exasol#spark-connector_2.12;1.2.0: not found

 

Please find attached screenshot:

achu310197_0-1643892859752.png

 

Kindly help me resolving the issue,

What can be the possible package and class that need to be given while spark-submit in --packages --class respectively?

1 REPLY 1

exa-Muhammet
Team Exasol
Team Exasol

Hello,

Since you are already providing the assembled jar file using --jars configuration parameter, you could remove the --packages parameter. Either one of them should be provided.

 

The packages parameter looks for the connector artifact in the Maven repositories. If you only want to use packages parameter, please update the version which seems wrong in your case.

--packages com.exasol:spark-connector_2.12:1.2.0-spark-2.4.8

With --packages you should also provide Exasol repository:

--repositories https://maven.exasol.com/artifactory/exasol-releases

Hope this help!

Best