Remotely execute program in multiple EC2 instances and wait for results

Question

Suppose there are three EC2 instances called A,X,Y.

I'd like to execute Java program jx, jy in X, Y from other Java program ja in A. ja needs to pass command line argument of jx and jy.

For EC2 instance A, X, Y, and Java program ja, jx, jy, 
A's ja------------> X's jx
     -------------> Y's jy

The ja wait for finish of both jx and jy and obtain the result value (or file)

A's ja <---return val -------X's jx
       <---return val ------ Y's jy

Again, ja execute jx and jy with other argument determined by the return values.

ja needs to execute jx and jy for 5 times respectively.

I think psudo code of ja is like below:

for(int i = 0; i < 5; i++){
  rx = executeJX(argx);
  ry = executeJY(argy);

  waitJxJyFinish(); // jx and jy take ~30 min to finish.
  (argx, argy) = determineNextArgXandArgY(rx, ry);
}

Constraint

This is for just experimental purpose, not production for daily use, that's why the least overhead way is the best even if it's dirty way. I don't have any experience of hadoop or any other staff. Simple and easy way is welcome.

user1832464 · Accepted Answer · 2015-02-16 11:16:28Z

1

IF you are looking to pass variables between instances in an asynchronous way to run tasks, I think the best solution would be an Amazon SQS Queue.

This would allow you to run the Java Program on X and Y, and submit the result to the queue as a message containing the result that you want to use on A. A can then intermittently poll the queue for the message, which contains the result, and then run jA using those variables.

This is the most elegant solution and decouples your application. Another alternative would be to post the results from X and Y to S3 as a file, which A then occasionally checks for and takes a copy of if it exists.

answered Feb 16, 2015 at 11:16

user1832464

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Remotely execute program in multiple EC2 instances and wait for results

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related