AWS - Lambda cannot access Livy endpoint for EMR

Question

For reference, livy is a rest endpoint used to pull data from a cluster. Within the same account, my lambda function always times out when attempting to access the livy endpoint using by EMR.

Endpoint: http://ip-xxxx-xx-xxx-xx.ec2.internal:8998/

Error:

     "errorMessage": "HTTPConnectionPool(host='ip-xxx-xx-xxxx-xx.ec2.internal', port=`8998`):
Max retries exceeded with url: /batches (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa6613150d0>: Failed to establish a new connection: [Errno 110] Connection timed out'))",

I added AmazonEMRFullAccessPolicy_v2 to the lambda, but it did not help. What am I missing? Is this the right endpoint to use for internal access?

SnigJi · Accepted Answer · 2021-07-16 18:46:42Z

1

Firstly, your http endpoint is <ip>.ec2.internal. So Im assuming its in some VPC (not public). So your lambda must be in within the same VPC.

You are trying to access HTTP endpoint with a TCP connection, so your lambda security group must be whitelisted to EMR master node's security group.

Lastly, You don't need AmazonEMRFullAccessPolicy_v2 because you are not accessing AWS API.

answered Jul 16, 2021 at 18:46

SnigJi

1,41811 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

AWS - Lambda cannot access Livy endpoint for EMR

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related