AWS : How to guarantee availability of static Private IP while bootstrapping

Question

I am launching EMR cluster and while bootstrapping I am assigning a Private IP address to it master node. So this get associated when cluster starts and gets released when cluster gets terminated. How do i guarantee that this IP is always available for my cluster and no other process picks it up even when not in use.

Once I get this Ip, I submit my spark jars to 8998 port of this IP, using Apache livy REST API

So my use case is to expose the IP on master node, so that LIVY APIs can submit job to EMR.

Could you please tell us more about your use-case? For example, why do you want a static private IP address? There might be another way to achieve your objective (eg using DNS names), but you'll need to tell us more. Please edit your question to add details rather than adding a comment. — John Rotenstein
– John Rotenstein, Commented Jun 8, 2020 at 6:32
You can create a network interface and attach it to master node. But you need to configure other services (like yarn, hadoop, etc.) to work with the 2ndary network interface. Once the cluster is terminated your interface will automatically gets detached. — SnigJi
– SnigJi, Commented Jun 8, 2020 at 7:56

John Rotenstein · Accepted Answer · 2020-06-08 06:58:13Z

2

It is not possible to request a specific private IP address for the cluster.

Instead, I would suggest:

Create a Route 53 private hosted zone with a Record Set for the cluster (eg cluster.private)
After launching the EMR cluster, update the Record Set for the IP address of the master node
Have Apache Livy reference the cluster by DNS Name rather than IP address

Updating the Record Set can be done programmatically. A script would do something like:

Call list_clusters()
For each cluster, call describe_cluster()
Look for a tag (eg DNS=cluster.private)
If the DNS tag is present:
- Update the named Record Set in Route 53 to use an A-Record to point to the IP address of the master node

This script would need to be triggered after the cluster is launched.

answered Jun 8, 2020 at 6:58

John Rotenstein

273k28 gold badges456 silver badges541 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

manisha-sh Over a year ago

Thanks much. I will try this out and post here.

Collectives™ on Stack Overflow

AWS : How to guarantee availability of static Private IP while bootstrapping

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related