I have the standard (simplified) pattern, in a step machine:
- start a glue job
- get job satus
- if status is running:
- wait
- go back to get job status
My issue is with the connection between the wait state and the get job status state. I cannot forge an output which mimics the output of the starting job.
This is my current setup:
{
"StartAt": "Parallel",
"QueryLanguage": "JSONPath",
"States": {
"Parallel": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "Start Glue Job",
"States": {
"Start Glue Job": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun",
"Parameters": {
"JobName": "job_name"
},
"Next": "Get status"
},
"Get status": {
"Type": "Task",
"Parameters": {
"JobName.$": "$.JobName",
"RunId.$": "$.JobRunId"
},
"Resource": "arn:aws:states:::aws-sdk:glue:getJobRun",
"Next": "Choice"
},
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.JobRun.JobRunState",
"StringEquals": "RUNNING",
"Next": "Wait"
}
],
"Default": "Complete"
},
"Wait": {
"Type": "Wait",
"Seconds": 1,
"QueryLanguage": "JSONata",
"Next": "Get status",
"Output": {
"JobName": "{% $.JobRun.JobName %}", // THIS WHERE I HAVE AN ISSUE!
"JobRunId": "{% $.JobRun.Id %}"
}
},
"Complete": {
"Type": "Pass",
"End": true
}
}
},
{
"StartAt": "Placeholder Pass",
"States": {
"Placeholder Pass": {
"Type": "Pass",
"End": true
}
}
}
],
"Next": "Success",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "Fail"
}
]
},
"Fail": {
"Type": "Fail"
},
"Success": {
"Type": "Succeed"
}
}
}
An execution tells me:
{
"Error": "Glue.EntityNotFoundException",
"Cause": "Job run not found for the given job name and runId. (Service: Glue, Status Code: 400, Request ID: c8ab4d3e-16ce-4fcd-8fe3-96a1aecb6bce)"
}
How can I pass output from wait in the expected format?
For the context, I am trying to have parallel glue jobs, which will complete even if one fails. If there is another direction I should look at, links are welcome.
