Session isn't active Pyspark in an AWS EMR cluster

Question

I have opened an AWS EMR cluster and in pyspark3 jupyter notebook I run this code:

"..
textRdd = sparkDF.select(textColName).rdd.flatMap(lambda x: x)
textRdd.collect().show()
.."

I got this error:

An error was encountered:
Invalid status code '400' from http://..../sessions/4/statements/7 with error payload: {"msg":"requirement failed: Session isn't active."}

Running the line:

sparkDF.show()

works!

I also created a small subset of the file and all my code runs fine.

What is the problem?

Wait for a while the notebook creates a session to the EMR or restart kernel. just timeout I think — Daeho Ro
– Daeho Ro, Commented Sep 23, 2019 at 12:53
the cluster is open for two hours now, how long do I need to wait? why I don't need to wait for the small subset? — anat
– anat, Commented Sep 23, 2019 at 12:58
Not cluster but your notebook. Check the application log for your EMR that the livy session by notebook is working well. — Daeho Ro
– Daeho Ro, Commented Sep 23, 2019 at 13:01
Your EMR console > application history and find livy-session-xx for numbering xx like 1, 2, ... — Daeho Ro
– Daeho Ro, Commented Sep 23, 2019 at 13:09

Koba · Accepted Answer · 2019-10-25 16:20:52Z

22

I had the same issue and the reason for the timeout is the driver running out of memory. Since you run collect() all the data gets sent to the driver. By default the driver memory is 1000M when creating a spark application through JupyterHub even if you set a higher value through config.json. You can see that by executing the code from within a jupyter notebook

spark.sparkContext.getConf().get('spark.driver.memory')

1000M

To increase the driver memory just do

%%configure -f 
{"driverMemory": "6000M"}

This will restart the application with increased driver memory. You might need to use higher values for your data. Hope it helps.

answered Oct 25, 2019 at 16:20

Koba

1,5444 gold badges28 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ijoseph Over a year ago

IMO the driver could have died for any number of reasons. However, the %%configure -f command will restart it, regardless.

Conso Over a year ago

For me worked: %%configure -f {"spark.driver.memory": "6000M"} and not "driverMemory"

Nithish Inpursuit Ofhappiness · Accepted Answer · 2019-12-23 21:31:12Z

10

From This stack overflow question's answer which worked for me

Judging by the output, if your application is not finishing with a FAILED status, that sounds like a Livy timeout error: your application is likely taking longer than the defined timeout for a Livy session (which defaults to 1h), so even despite the Spark app succeeds your notebook will receive this error if the app takes longer than the Livy session's timeout.

If that's the case, here's how to address it:

1. edit the /etc/livy/conf/livy.conf file (in the cluster's master node)
2. set the livy.server.session.timeout to a higher value, like 8h (or larger, depending on your app)
3. restart Livy to update the setting: sudo restart livy-server in the cluster's master
4. test your code again

Alternative way to edit this setting - https://allinonescript.com/questions/54220381/how-to-set-livy-server-session-timeout-on-emr-cluster-boostrap

answered Dec 23, 2019 at 21:31

Nithish Inpursuit Ofhappiness

1,7213 gold badges19 silver badges40 bronze badges

1 Comment

Shashank Over a year ago

Thanks for the suggestion. I found the config for Livy timeout that can be passed to the EMR cluster as a JSON file here: stackoverflow.com/a/54240619/4306852

Nandish Madhu · Accepted Answer · 2021-02-25 03:36:19Z

2

Just a restart helped solve this problem for me. On your Jupyter Notebook, go to -->Kernel-->>Restart Once done, if you run the cell with "spark" command you will see that a new spark session gets established.

answered Feb 25, 2021 at 3:36

Nandish Madhu

211 bronze badge

Comments

Fabio Manzano · Accepted Answer · 2019-09-25 20:49:53Z

0

You might get some insights from this similar Stack Overflow thread: Timeout error: Error with 400 StatusCode: "requirement failed: Session isn't active."

Solution might be to increase spark.executor.heartbeatInterval. Default is 10 seconds.

See EMR's official documentation on how to change Spark defaults:

You change the defaults in spark-defaults.conf using the spark-defaults configuration classification or the maximizeResourceAllocation setting in the spark configuration classification.

answered Sep 25, 2019 at 20:49

Fabio Manzano

2,8651 gold badge13 silver badges23 bronze badges

2 Comments

anat Over a year ago

Thank you, I'll try it and let you know.

Koba Over a year ago

I tried increasing the heartbeatInterval up to 110s and it didnt solve the issue for me. The livy session would get always disconnected. I am running the code in a jupyter notebook, however, if I run the same code with spark-submit it works no problem.

Manoj Raja Rao · Accepted Answer · 2019-10-05 06:48:17Z

0

Insufficient reputation to comment.

I tried increasing heartbeat Interval to a much higher (100 seconds), still the same result. FWIW, the error shows up in < 9s.

answered Oct 5, 2019 at 6:48

Manoj Raja Rao

551 silver badge6 bronze badges

Comments

Elhanan Mishraky · Accepted Answer · 2020-05-21 09:38:17Z

0

What worked for me is adding {"Classification": "spark-defaults", "Properties": {"spark.driver.memory": "20G"}} to the EMR configuration.

answered May 21, 2020 at 9:38

Elhanan Mishraky

2,82627 silver badges26 bronze badges

Collectives™ on Stack Overflow

Session isn't active Pyspark in an AWS EMR cluster

6 Answers 6

2 Comments

1 Comment

Comments

2 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

2 Comments

1 Comment

Comments

2 Comments

Comments

Comments

Linked

Related