Airflow Dataproc operator to run shell scripts

Question

Is there any direct way to run shell scripts into dataproc cluster. currently i can run the shells through pysparkoperator (which calls aonther python file and then this python file calls shell script). I have searched many links but as of now not found any direct way .

It will be really helpful for me if anybody can tell me the easiest way.

I don't know a direct way, but in case you are not aware, you can 1)find the running Dataproc master node Name 2)gcloud compute ssh to that instance — Chengzhi
– Chengzhi, Commented Sep 19, 2017 at 14:06
Hi , yes using this option we can work on shell scripts manually but my objective is to schedule this script through airflow . Let me know if you can help me more. Thanks in advance — Adt
– Adt, Commented Sep 20, 2017 at 6:49
All with PythonOperator, You can use googleapiclient.discovery.build('dataproc', 'v1', credentials=GoogleCredentials.get_application_default()) to get the running dataproc, then subprocess.Popen with gcloud compute ssh by passing the correct instance name — Chengzhi
– Chengzhi, Commented Sep 20, 2017 at 18:10

tix · Accepted Answer · 2017-09-19 18:26:04Z

1

PIG job with sh operator [1]: gcloud dataproc jobs submit pig ... -e 'sh ls'

I am however curious what the end goal is? Why run shell scripts? If your intent is to perform one-time cluster setup then you should use initialization actions [2].

[1] https://pig.apache.org/docs/r0.9.1/cmds.html#sh

[2] https://cloud.google.com/dataproc/docs/concepts/init-actions

answered Sep 19, 2017 at 18:26

tix

2,13314 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ashish Kumar · Accepted Answer · 2018-12-23 04:40:18Z

0

You can use airflow BashOperator and use the following command:

gcloud compute ssh user@server --zone your_cluster_zone \
  --command='Your Command'

Example:

    BashCommand= BashOperator(
    task_id='BashCommand',
    bash_command='gcloud compute ssh user@server --zone your_cluster_zone --command='Your Command',
    dag=dag)

answered Dec 23, 2018 at 4:40

Ashish Kumar

5911 gold badge5 silver badges22 bronze badges

Collectives™ on Stack Overflow

Airflow Dataproc operator to run shell scripts

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related