• Register
0 votes
207 views

Problem :

I have my Spark app which runs without any problem in the local mode, but have some problems when trying to submit to my Spark cluster.

The error message is as below:

 

20/01/05 15:42:06 WARN scheduler.TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2, cluster-node-02): java.lang.ExceptionInInitializerError

    at GroupEvolutionES$$anonfun$6.apply(GroupEvolutionES.scala:579)

    at GroupEvolutionES$$anonfun$6.apply(GroupEvolutionES.scala:579)

    at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:390)

    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)

    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)

    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)

    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)

    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)

    at org.apache.spark.scheduler.Task.run(Task.scala:89)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration

    at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)

In the above error message, GroupEvolutionES is my main class. Above error message says "A master URL must be set in your configuration", but I have already provided the "--master" parameter to my spark-submit.

Anybody who can fix above problem?

My Spark version : 1.6.1

6 5 3
7,540 points

Please log in or register to answer this question.

1 Answer

0 votes

Solution :

The default value of the "spark.master" is always spark://HOST:PORT, and the below code tries to get the session from your standalone cluster that is running at the HOST:PORT, and is expecting the HOST:PORT value to be present in your spark config file.

SparkSession myspark = SparkSession
    .builder()
    .appName("SomeMyAppName")
    .getOrCreate();

Here "org.apache.spark.SparkException: A master URL must be set in your configuration" trying to say that HOST:PORT is not set in your spark configuration file.

To ignore the value of "HOST:PORT", you need to set myspark.master as local

SparkSession myspark = SparkSession
    .builder()
    .appName("SomeMyAppName")
    .config("myspark.master", "local")
    .getOrCreate();

 

9 7 4
38,600 points

Related questions

0 votes
1 answer 14 views
14 views
Problem: who know this problem please help? error: object apache is not a member of package org
asked Mar 30 Rohit kr 15.5k points
0 votes
1 answer 2 views
2 views
Problem Provide me the method to find number of executors in spark.
asked 5 days ago salim1 34.8k points
0 votes
1 answer 2 views
2 views
Problem Why is the error &ldquo;Unable to find encoder for type stored in a Dataset&rdquo; when encoding JSON using case classes?
asked 1 day ago neeraj 9.5k points
0 votes
1 answer 2.1K views
2.1K views
Problem : I want to view my php files locally on my mac. I am trying to run Yosemite on my mac. I have the MAMP installed on my mac. When I press start my MySQL server starts as usual but my Apache doesn't start at all. An error message is shown ... also tried reinstalling the OSX and that worked but until I shut down my mac the problem reoccurs. Can anyone help with the solution on above error?
asked Dec 19, 2019 alecxe 7.5k points
0 votes
1 answer 3 views
3 views
Problem: Looking for some help to solve this problem "Expressions in scala evaluated using method called? ".
asked 3 days ago zayed1 30.9k points
0 votes
1 answer 4 views
4 views
Problem What is the explanation or is there a resource to learn more about the Scala case classes from? Can anyone help me to solve it...
asked Mar 9 neeraj 9.5k points
0 votes
1 answer 2 views
2 views
Problem: I am using angular in my front-end and this is the piece of code where i call my loclhost , todo variable should have the value, but I get error: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access.
asked Apr 2 ummesalma 24.5k points
0 votes
1 answer 3 views
3 views
Problem: When running the Scala interpreter in Ubuntu 14.04, I get the following message printed as the first line: Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
asked Mar 31 ummesalma 24.5k points
0 votes
1 answer 7 views
7 views
Problem: How to do it? Java gateway process exited before sending the driver its port number
asked Mar 25 Ifra 24.4k points
0 votes
1 answer 440 views
440 views
Problem: I have recently started learning pyspark. I am very new to pyspark. I have written below code but it sowing me the error. from pyspark import SparkContext, SparkConf from pyspark.sql import SQLContext conf = SparkConf().setAppName("myApp").setMaster("local") sc ... 'ind', "state"]) Attributeerror: 'pipelinedrdd' object has no attribute 'todf' Can somebody help me in fixing my above code?
asked Aug 10, 2020 Raphael Pacheco 4.9k points