menu
  • Register

Java gateway process exited before sending its port number

2 votes
73 views

Problem:

I got this Spark connection issue, and SparkContext didn't work for sc.

The command to initialize ipython notebook:

ipython notebook --profile=pyspark

Environment:
Mac OS
Python 2.7.10
Spark 1.4.1
java version "1.8.0_65"

Can anyone explain or help?

●10 ●7 ●2
6,060 points

1 Answer

2 votes

Answer:

You actually have to define "pyspark-shell" in PYSPARK_SUBMIT_ARGS if you define this.

For instance:

import os

os.environ['PYSPARK_SUBMIT_ARGS'] = "--master mymaster --total-executor 2 --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.mycorp.com-Dhttp.proxyPort=1234 -Dhttp.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 -Dhttps.proxyHost=proxy.mycorp.com -Dhttps.proxyPort=1234 -Dhttps.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 pyspark-shell"

 

●11 ●6 ●4
34,950 points

Related questions

1 vote
1 answer 283 views
283 views
Java gateway process exited before sending its port number
Problem: I got this Spark connection issue, and SparkContext didn't work for sc. The command to initialize ipython notebook: ipython notebook --profile=pyspark Environment: Mac OS Python 2.7.10 Spark 1.4.1 java version "1.8.0_65" Can anyone help?
asked Mar 27 LizzyM 6.1k points
0 votes
1 answer 84 views
84 views
java gateway process exited before sending the driver its port number
Problem : I am getting bellow error while trying to run pyspark. java gateway process exited before sending the driver its port number
asked Oct 31, 2019 peterlaw 6.9k points
0 votes
2 answers 531 views
531 views
exception: java gateway process exited before sending the driver its port number
Problem : I am getting bellow error while trying to run pyspark on my macbook air exception: java gateway process exited before sending the driver its port number
asked Oct 19, 2019 peterlaw 6.9k points
0 votes
1 answer 115 views
115 views
Attributeerror: 'pipelinedrdd' object has no attribute 'todf'.
Problem: I have recently started learning pyspark. I am very new to pyspark. I have written below code but it sowing me the error. from pyspark import SparkContext, SparkConf from pyspark.sql import SQLContext conf = SparkConf().setAppName("myApp").setMaster("local") sc ... 'ind', "state"]) Attributeerror: 'pipelinedrdd' object has no attribute 'todf' Can somebody help me in fixing my above code?
asked Aug 10 Raphael Pacheco 4.9k points
Dark theme