1

I'm experiencing a strange issue with an AWS Lambda function that uses a Docker image stored in Amazon ECR. The function has an alias named stable pointing to a published version.

Observed behavior:

  • Everything works fine right after deployment.
  • After a few hours (typically 6–24), the Lambda invoked via the stable alias starts failing with:
CodeArtifactUserFailedException: Failed to restore the function xxx: The function does not have permission to access the specified image.
  • The AWS Console also shows the message: "Failed to restore the function xxx: The function does not have permission to access the specified image."

  • If I create a new version of the Lambda (with the same image and same configuration and same role) and invoke $LATEST, it works perfectly.

Context:

  • The image is stored in a private ECR repository.

  • There’s a lifecycle policy in place to retain only the last 5 images.

  • A Lambda warmer runs every 5 minutes to prevent cold starts.

  • The Lambda has the standard permissions (AWSLambdaBasicExecutionRole + AmazonEC2ContainerRegistryReadOnly).

  • The lambda becomes inactive:

"State": "Inactive",
"StateReason": "The function does not have permission to access the specified image.",
"StateReasonCode": "ImageAccessDenied",

Hypotheses:

  1. Could it be that the published Lambda version points to an ECR image digest that gets deleted by the lifecycle policy, making it inaccessible?
  2. But if that’s the case, why does $LATEST still work fine using the same image?
  3. The pulled image corrupts and the warmer make lambda service unable to get rid of corrupted ones, we disable it, but the stable alias still not works.
2
  • May this issue be related to this answer? stackoverflow.com/a/75376559 Commented Aug 1 at 6:05
  • No, we change the tag every new deploy. The issue is not triggered by a deploy Commented Aug 4 at 7:41

2 Answers 2

0

We opened a Support Request to AWS and seems that if you make changes to ECR repository policy or IAM Policy, you must redeploy the lambda.

In our case seems that CloudFormation made a DeleteRepositoryPolicy action that causes the loss of permission.

Even if you restore the permission, seems have no effects.

Sign up to request clarification or add additional context in comments.

Comments

0

Looking at the AWS Lambda documentation, it seems that the error is related to some optimization errors: https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-invocation.html#troubleshooting-deployment-container-artifact

Error: CodeArtifactUserFailedException error message

Lambda failed to optimize the code. You need to correct the code and upload it again. HTTP response code 409.

1 Comment

Probably you missi this: If I create a new version of the Lambda (with the same image and same configuration and same role) and invoke $LATEST, it works perfectly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.