Execute shell script using google cloud dataflow job

Question

I am trying to execute shell script placed in a bucket using dataflow job. I can execute gsutil commands using this job using Direct Runner :

String[] cmdline = { "cmd.exe", "/c", "gsutil ls gs://Bucketname" };
Process p = Runtime.getRuntime().exec(cmdline); 
BufferedReader reader = new BufferedReader(new 
InputStreamReader(p.getInputStream()));
           String line = null;
           while ((line = reader.readLine()) != null)
           {
              System.out.println(line);
           }

Note : I will use dataflow runner to execute the script because i am using windows machine.

Are you trying to run this from within a ParDo? Is this failing? If so, what error are you seeing? What does your pipeline look like? — Pablo
– Pablo, Commented Aug 23, 2017 at 0:03
If i write String command = "bash shellDataflow.sh"; Process p = Runtime.getRuntime().exec(command);. I get error message as " execption occured Cannot run program "bash": CreateProcess error=2, The system cannot find the file specified" . Error is like this because above program is invoking my windows system's command line instance . Is there any way to connect to cloud console enviroment through java code so that i can run my shell in that enviroment using my dataflow job. — Adt
– Adt, Commented Aug 23, 2017 at 10:54

Vahid Boreiri · Accepted Answer · 2017-09-21 06:00:20Z

1

Try using this. It runs in my case. You have to deploy the code in cloud as a .jar or a maven project. The path /home/*/test.sh is in the cloud console.

String[] cmd = {"sh",  "/home/akash/test.sh", "/home/akash/"};
        Runtime.getRuntime().exec(cmd);

edited Sep 21, 2017 at 6:00

Vahid Boreiri

3,4561 gold badge21 silver badges35 bronze badges

answered Sep 21, 2017 at 5:44

Arkanil Dutta

451 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Execute shell script using google cloud dataflow job

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related