• Register
1 vote
636 views

Problem:

I got this Spark connection issue, and SparkContext didn't work for sc.

The command to initialize ipython notebook:

ipython notebook --profile=pyspark

Environment:
Mac OS
Python 2.7.10
Spark 1.4.1
java version "1.8.0_65"

Can anyone help?

10 7 2
6,060 points

Please log in or register to answer this question.

1 Answer

1 vote

Answer:

You actually have to define

"pyspark-shell" in PYSPARK_SUBMIT_ARGS

if you define this.

For instance:

import os

os.environ['PYSPARK_SUBMIT_ARGS'] = "--master mymaster --total-executor 2 --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.mycorp.com-Dhttp.proxyPort=1234 -Dhttp.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 -Dhttps.proxyHost=proxy.mycorp.com -Dhttps.proxyPort=1234 -Dhttps.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 pyspark-shell"

 

11 6 4
34,950 points

Related questions

0 votes
1 answer 20 views
20 views
Problem: How to do it? Java gateway process exited before sending the driver its port number
asked Mar 25 Ifra 36.4k points
0 votes
1 answer 161 views
161 views
Problem : I am getting bellow error while trying to run pyspark. java gateway process exited before sending the driver its port number
asked Oct 31, 2019 peterlaw 6.9k points
0 votes
2 answers 1.2K views
1.2K views
Problem : I am getting bellow error while trying to run pyspark on my macbook air exception: java gateway process exited before sending the driver its port number
asked Oct 19, 2019 peterlaw 6.9k points
2 votes
1 answer 132 views
132 views
Problem: I got this Spark connection issue, and SparkContext didn't work for sc. The command to initialize ipython notebook: ipython notebook --profile=pyspark Environment: Mac OS Python 2.7.10 Spark 1.4.1 java version "1.8.0_65" Can anyone explain or help?
asked Mar 24, 2020 LizzyM 6.1k points
0 votes
1 answer 275 views
0 votes
1 answer 11 views
11 views
Problem: I need your help; exception: java gateway process exited before sending its port number
asked Mar 19 Wafa Abu Yousef 6.1k points
0 votes
1 answer 13 views
13 views
Solution: I need help in this: exception: java gateway process exited before sending its port number
asked Mar 17 Wafa Abu Yousef 6.1k points
0 votes
1 answer 508 views
508 views
Problem: I have recently started learning pyspark. I am very new to pyspark. I have written below code but it sowing me the error. from pyspark import SparkContext, SparkConf from pyspark.sql import SQLContext conf = SparkConf().setAppName("myApp").setMaster("local") sc ... 'ind', "state"]) Attributeerror: 'pipelinedrdd' object has no attribute 'todf' Can somebody help me in fixing my above code?
asked Aug 10, 2020 Raphael Pacheco 4.9k points
0 votes
1 answer 3 views
3 views
Problem: I am trying to leverage spark partitioning. I was trying to do something like data.write.partitionBy("key").parquet("/location") The issue here each partition creates huge number of parquet files which result slow read if I am trying to read from the ... partition coalesce to a certain number and store at a separate location. How should I use partitioning to avoid many files after write?
asked Apr 30 muktaa 34.6k points