Starting an AWS Glue job from Lambda using Python?

Question

I am attempting to start an AWS Glue job (Python shell) via a Lambda when a new file is dropped into an S3 bucket. I have the Glue job setup and it operates as expected if I run manually. I thought that triggering the Glue job from a Lambda triggered by the S3 create would be simple. So far I have the Lambda created and it does run when the S3 file is created, however it does not want to actually start the Glue job, or provide any feedback as to why it won't start. Below is the Python 3.8 code I have been using in my Lambda:

import boto3
from botocore.exceptions import ClientError

def handler(event, context):
   glue_client = boto3.client('glue')
   job_name = 'my-glue-job-name'
   
   try:
      print('Attempting to start glue job:', job_name)
      job_run_id = glue_client.start_job_run(JobName=job_name)
      print('Running Glue job, id:', job_run_id)
      return job_run_id
   except ClientError as e:
      print('>>>>>error 1:', e)
      raise Exception( "boto3 client error in run_glue_job: " + e.__str__())
   except Exception as e:
      print('>>>>>error 2:', e)
      raise Exception( "Unexpected error in run_glue_job: " + e.__str__())

When i check the Lambda's logs I can see the lambda started when the file was created in S3. I can see the print entry 'Attempting to start glue job: my-glue-job-name'. And thats all I see. I don't see 'Running Glue job, id: xxx' printed log entry, nor do I see any error messages. Likewise the Glue job logs show no indication of being started.

I have given the Lambda the AWSGlueServiceRole policy so I don't think its a permissions issue.

Any ideas are appreciated.

Bruno Schaatsbergen · Accepted Answer · 2021-11-28 17:17:47Z

1

Could you share the service-linked role you've setup and attached to the function with us?

By reading through the boto3 Glue spec and the AWS docs the following is required:

An AWS Identity and Access Management (IAM) role for Lambda with permission to run AWS Glue jobs. For example, set up a service-linked role for Lambda that has the AWSGlueServiceRole policy attached to it.

answered Nov 28, 2021 at 17:17

Bruno Schaatsbergen

4165 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Gatmando Over a year ago

Thanks Bruno, that was a legit avenue to investigate but turns out that wasn't the issue. It was VPC issue.

Bruno Schaatsbergen Over a year ago

Grea that you found it!

Gatmando · Accepted Answer · 2021-11-29 18:55:57Z

0

Turns out it wasn't an IAM permissions issue at all but instead a VPC issue. All of our Glue service need a VPC endpoint added to allow access from other services in our account. Once that was done it worked as expected.

edited Nov 29, 2021 at 18:55

answered Nov 29, 2021 at 17:07

Gatmando

2,2893 gold badges32 silver badges65 bronze badges

Collectives™ on Stack Overflow

Starting an AWS Glue job from Lambda using Python?

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related