connect local python script to remote spark master

Question

I am using python 2.7 with spark standalone cluster.

When I start the master on the same machine running the python script. it works smoothly.

When I start the master on a remote machine, and try to start spark context on the local machine to access the remote spark master. nothing happens and i get a massage saying that the task did not get any resources.

When i access the master's UI. i see the job, but nothing happens with it, it's just there.

How do i access a remote spark master via a local python script?

Thanks.

EDIT: I read that in order to do this i need to run the cluster in cluster mode (not client mode), and I found that currently standalone mode does not support this for python application.

Ideas?

thebeancounter · Accepted Answer · 2017-08-29 07:57:32Z

3

in order to do this i need to run the cluster in cluster mode (not client mode), and I found here that currently standalone mode does not support this for python application.

answered Aug 29, 2017 at 7:57

thebeancounter

4,85911 gold badges72 silver badges121 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

connect local python script to remote spark master

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related