Jupyter Notebook error while using PySpark Kernel: the code failed because of a fatal error: Error sending http request

Question

I and using jupyter notebook's PySpark kernel, I have successfully selected PySpark kernel but I keep getting the below error

The code failed because of a fatal error: Error sending http request and maximum retry encountered.. Some things to try:

a) Make sure Spark has enough available resources for Jupyter to create a Spark context.

b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.

c) Restart the kernel.

here's the log also

2019-10-10 13:37:43,741 DEBUG   SparkMagics Initialized spark magics.
2019-10-10 13:37:43,742 INFO    EventsHandler   InstanceId: 32a21583-6879-4ad5-88bf-e07af0b09387,EventName: notebookLoaded,Timestamp: 2019-10-10 10:37:43.742475
2019-10-10 13:37:43,744 DEBUG   python_jupyter_kernel   Loaded magics.
2019-10-10 13:37:43,744 DEBUG   python_jupyter_kernel   Changed language.
2019-10-10 13:37:44,356 DEBUG   python_jupyter_kernel   Registered auto viz.
2019-10-10 13:37:45,440 INFO    EventsHandler   InstanceId: 32a21583-6879-4ad5-88bf-e07af0b09387,EventName: notebookSessionCreationStart,Timestamp: 2019-10-10 10:37:45.440323,SessionGuid: d230b1f3-6bb1-4a66-bde1-7a73a14d7939,LivyKind: pyspark
2019-10-10 13:37:49,591 ERROR   ReliableHttpClient  Request to 'http://localhost:8998/sessions' failed with 'HTTPConnectionPool(host='localhost', port=8998): Max retries exceeded with url: /sessions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000013184159808>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))'
2019-10-10 13:37:49,591 INFO    EventsHandler   InstanceId: 32a21583-6879-4ad5-88bf-e07af0b09387,EventName: notebookSessionCreationEnd,Timestamp: 2019-10-10 10:37:49.591650,SessionGuid: d230b1f3-6bb1-4a66-bde1-7a73a14d7939,LivyKind: pyspark,SessionId: -1,Status: not_started,Success: False,ExceptionType: HttpClientException,ExceptionMessage: Error sending http request and maximum retry encountered.
2019-10-10 13:37:49,591 ERROR   SparkMagics Error creating session: Error sending http request and maximum retry encountered.

note that I am trying to configure this on windows. thanks alot

Sjoerd · Accepted Answer · 2021-08-05 09:20:16Z

8

I faced the same issue, you can solve it by not using a PySpark Kernel (notebook) but a Python 3 kernel (notebook). I used the following code to setup the Spark cluster:

import pyspark # only run after findspark.init()
from pyspark.sql import SparkSession
# May take awhile locally
spark = SparkSession.builder.appName("test").getOrCreate()
spark

answered Aug 5, 2021 at 9:20

Sjoerd

1111 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jstark Over a year ago

wow, timely answer, this just fixed it for me. Is this a hack? Should we be able to use the PySpark kernel?

Sjoerd Over a year ago

Good question, I quickly checked online, but I don't see a lot of other people facing the same issue. I think most people are just using the Python kernel.

prashant thakre Over a year ago

That great answer, I wasted my 3 hours. Your answer saved my full day.

Luis Leal Over a year ago

In my case, this solves the issue because it is not connecting to the cluster, just runs a local spark session, did you any config to some file?

Luis Galvez · Accepted Answer · 2020-01-11 19:33:40Z

0

If you are trying to connect your Jupyter Notebook to a Spark server through Livy (e.g. AWS Glue Development Endpoint), you have to replace "localhost" with the Spark server IP address in: ~/.sparkmagic/config.json

As mentioned here: https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/

answered Jan 11, 2020 at 19:33

Luis Galvez

4914 silver badges12 bronze badges

1 Comment

Sowmya Over a year ago

Even after following the above link I am too getting the similar issue.

AB_87 · Accepted Answer · 2021-10-13 23:38:59Z

-1

Posting below answer as it may help someone facing this issue when using sagemaker notebook with Glue Dev Endpoint.

I received same error message in my PySpark kernel notebook. In my case issue was missing Lifecycle configuration attached to notebook instance which was somehow removed. I delete and recreate dev endpoint every day but it lifecycle config normally remains attached to notebook.

answered Oct 13, 2021 at 23:38

AB_87

1,1568 silver badges18 bronze badges

Collectives™ on Stack Overflow

Jupyter Notebook error while using PySpark Kernel: the code failed because of a fatal error: Error sending http request

3 Answers 3

4 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related