I am looking to deploy my machine learning model to production. I am using API Gateway + Lambda to trigger inferencing on an EC2 instance. The inferencing takes a while (~30 secs). The Lambda timeout is set to 2 minutes.
When I use Postman or my browser to call the HTTP API, I am getting a 503 Service Unavailable error in 5 secs. At the backend I am seeing that the request is successful and success response is sent back to the Lambda (after about 30 secs). I have read that max API Gateway timeout is 30 secs. I am wondering why I am getting a timeout in 5 secs then?
I am following instructions given here to deploy my model: https://francescopochetti.com/deploying-a-pretrained-gpt-2-model-on-aws/#Deploying_with_Lambda_EC2_and_DynamoDB
The lambda code is similar to given here: https://github.com/gabrielelanaro/ml-prototypes/blob/master/prototypes/styletransfer/huggingface/hugging_lambda.py.
If it helps, you can invoke the API using this URL: https://eekq9x0azg.execute-api.ap-south-1.amazonaws.com/gpt2?prompt=%20%22Nikhil%22&num_samples=3&length=60&temperature=0.7&top_p=0.9&top_k=40
Lambda logs in Cloudwatch:
2020-03-18T09:56:51.520+05:30 START RequestId: 59129756-d233-4bd5-8059-4087bd9f47e3 Version: $LATEST
2020-03-18T09:56:51.524+05:30
EVENT
2020-03-18T09:56:51.704+05:30 ['cd /home/ubuntu', 'shutdown -h +15', 'sudo -i -u ubuntu bash <<-EOF', 'source ~/.bashrc', 'source env/bin/activate', 'python3 gpt2-tweets.py --prompt=" "Nikhil"" --dynamoid=189377 --num_samples=3 --length=60 --temperature=0.7 --top_p=0.9 --top_k=40']
2020-03-18T09:57:22.051+05:30 The query returned the following items:
2020-03-18T09:57:22.051+05:30 [{'id': Decimal('189377'), 'text': '" Nikhil ji:\n\" I don't know, what do you want to say to the nation? I am in the UK. But I don't know what you are talking about. Sorry, that's not the point. I don t know what you want to say to the nation. I am"'}]
2020-03-18T09:57:22.091+05:30 END RequestId: 59129756-d233-4bd5-8059-4087bd9f47e3
2020-03-18T09:57:22.091+05:30 REPORT RequestId: 59129756-d233-4bd5-8059-4087bd9f47e3 Duration: 30567.70 ms Billed Duration: 30600 ms Memory Size: 128 MB Max Memory Used: 100 MB
