menu
  • Register

Java gateway process exited before sending its port number

2 votes
132 views

Problem:

I got this Spark connection issue, and SparkContext didn't work for sc.

The command to initialize ipython notebook:

ipython notebook --profile=pyspark

Environment:
Mac OS
Python 2.7.10
Spark 1.4.1
java version "1.8.0_65"

Can anyone explain or help?

●10 ●7 ●2
6,060 points

Please log in or register to answer this question.

1 Answer

2 votes

Answer:

You actually have to define "pyspark-shell" in PYSPARK_SUBMIT_ARGS if you define this.

For instance:

import os

os.environ['PYSPARK_SUBMIT_ARGS'] = "--master mymaster --total-executor 2 --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.mycorp.com-Dhttp.proxyPort=1234 -Dhttp.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 -Dhttps.proxyHost=proxy.mycorp.com -Dhttps.proxyPort=1234 -Dhttps.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 pyspark-shell"

 

●11 ●6 ●4
34,950 points

Related questions

0 votes
1 answer 20 views
20 views
Java gateway process exited before sending the driver its port number
Problem: How to do it? Java gateway process exited before sending the driver its port number
asked Mar 25 Ifra 36.4k points
1 vote
1 answer 636 views
636 views
Java gateway process exited before sending its port number
Problem: I got this Spark connection issue, and SparkContext didn't work for sc. The command to initialize ipython notebook: ipython notebook --profile=pyspark Environment: Mac OS Python 2.7.10 Spark 1.4.1 java version "1.8.0_65" Can anyone help?
asked Mar 27, 2020 LizzyM 6.1k points
0 votes
1 answer 161 views
161 views
java gateway process exited before sending the driver its port number
Problem : I am getting bellow error while trying to run pyspark. java gateway process exited before sending the driver its port number
asked Oct 31, 2019 peterlaw 6.9k points
0 votes
2 answers 1.2K views
1.2K views
exception: java gateway process exited before sending the driver its port number
Problem : I am getting bellow error while trying to run pyspark on my macbook air exception: java gateway process exited before sending the driver its port number
asked Oct 19, 2019 peterlaw 6.9k points
0 votes
1 answer 273 views
273 views
Pyspark: Exception: Java gateway process exited before sending the driver its port number
Please help! Thanks.
asked Jan 22 TeamScript 13.5k points
0 votes
1 answer 11 views
11 views
Exception: java gateway process exited before sending its port number .
Problem: I need your help; exception: java gateway process exited before sending its port number
asked Mar 19 Wafa Abu Yousef 6.1k points
0 votes
1 answer 13 views
13 views
exception: java gateway process exited before sending its port number.
Solution: I need help in this: exception: java gateway process exited before sending its port number
asked Mar 17 Wafa Abu Yousef 6.1k points
0 votes
1 answer 3 views
3 views
You need to create a large number of files and would like to ensure that you
Problem: I am trying to leverage spark partitioning. I was trying to do something like data.write.partitionBy("key").parquet("/location") The issue here each partition creates huge number of parquet files which result slow read if I am trying to read from the ... partition coalesce to a certain number and store at a separate location. How should I use partitioning to avoid many files after write?
asked Apr 30 muktaa 34.6k points
0 votes
1 answer 3 views
3 views
How to decide number of executors in spark ?
Problem Provide me the method to find number of executors in spark.
asked Apr 7 salim1 60.6k points
Dark theme