Lambda starts timing out randomly when communicating with DynamoDB

Question

I have a Node.js Lambda code base that talks to tiny dataset in DynamoDB (less than 400 byte each). Every now and then the function will time out over 5 minutes whilst doing a get() request to DynamoDB (via dynamoDbdAWS.DynamoDB.DocumentClient();).

The problem is it's completely random as to when this issue will occur but when it works it take ~2 second from a cold start, so taking over 5 minutes to run makes no sense and at random points.

It's a dev environment so only myself is using this, and I'm doing maybe 10 requests a day
context.callbackWaitsForEmptyEventLoop = false; has been set
Memory allocation never exceeds 45MB (128MB set)
I'm testing directly in Lambda
The code is deployed via Serverless
When testing, using Serverless, locally it works whilst the Lambda fails

I've inherited this project but have a good understanding of the architecture around it and it's fairly simple but I've not done much work with Lambda before.

Any ideas what I should look for or any known issues will be a massive help.

Is the Lambda function configured to run in a VPC?

Mark B
– Mark B

2019-01-09 13:00:25 +00:00
Commented Jan 9, 2019 at 13:00 — Mark B
– Mark B, Commented Jan 9, 2019 at 13:00
@MarkB Indeed it is.

WolfieZero
– WolfieZero

2019-01-09 13:11:09 +00:00
Commented Jan 9, 2019 at 13:11 — WolfieZero
– WolfieZero, Commented Jan 9, 2019 at 13:11

Mark B · Accepted Answer · 2019-01-09 13:32:43Z

2

It sounds like one (or more) of the VPC subnets the Lambda function is configured to run in doesn't have a route to a NAT Gateway (or an AWS PrivateLink configuration). So whenever that subnet is used by the Lambda function it is unable to access the AWS API.

If the Lambda function doesn't actually need to access any resources in the VPC then it is much better to not configure it to use the VPC at all.

answered Jan 9, 2019 at 13:32

Mark B

203k27 gold badges339 silver badges332 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

cementblocks Over a year ago

Adding a DynamoDB VPC endpoint would be a good solution, you can save on the NAT Gateway data cost.

WolfieZero Over a year ago

Looks like this could be the very issue. I'm going to have a look into this today and and hopefully confirm it is :)

WolfieZero Over a year ago

So we were using VPC Endpoints to connect to the DynamoDB instance but it turned out that there was an mis-configuration with the subnets as suggested. We had a public and private subnet attached rather than two privates.

Collectives™ on Stack Overflow

Lambda starts timing out randomly when communicating with DynamoDB

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related