AWS Step Function - passing Input to container override for AWS batch job

Question

I'm submitting an AWS Batch job from a Step Function. The batch job takes command line parameters. I can run it in the step function like this:

{
  "Comment": "Submit aws batch job with parameters",
  "StartAt": "SubmitJob",
  "States": {
    "SubmitJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName": "my-job",
        "JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:1",
        "JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
        "ContainerOverrides": {
          "Command": [
            "Hello",
            "World"
          ]
        }
      },
      "End": true
    }
  }
}

However, I would like to use the input to this step as the command line parameters. So I try to use the $.parameter notation in the Command.

My input is

{
  "param_1": "Hello",
  "param_2": "World"
}

and my step function is

{
  "Comment": "Submit aws batch job with parameters",
  "StartAt": "SubmitJob",
  "States": {
    "SubmitJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName": "my-job",
        "JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:1",
        "JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
        "ContainerOverrides": {
          "Command": [
            "$.param_1",
            "$.param_2"
          ]
        }
      },
      "End": true
    }
  }
}

However, this results in my batch job receiving the raw strings like "$.param_1" rather than them being substituted by the input "Hello". Why? What is required instead?

Scott · Accepted Answer · 2022-08-11 16:52:14Z

5

This is one Solution/Workaraound, I'm not sure if it is the only option.

In the AWS Batch Job Definition, in the Container properties, set Command to be

["Ref::param_1","Ref::param_2"]

These "Ref::" links will capture parameters that are provided when the Job is run. Additionally, you can specify parameters in the job definition Parameters section but this is only necessary if you want to provide defaults.

In the Step Functions State Machine definition, instead of using ContainerOverrides, use Parameters. Overall it will look like:

{
  "Comment": "Submit aws batch job with parameters",
  "StartAt": "SubmitJob",
  "States": {
    "SubmitJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName": "my-job",
        "JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:2",
        "JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
        "Parameters": {
          "param_1.$": "$.param_1",
          "param_2.$": "$.param_2"
        }
      },
      "End": true
    }
  }
}

Thus the input into the step function gets passed to the Batch job which captures them and uses as Command.

reference: https://docs.aws.amazon.com/batch/latest/userguide/example-job-definitions.html

answered Aug 11, 2022 at 16:52

Scott

1751 gold badge2 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Scott Over a year ago

Additionally, as long as you put the Parameters as in this answer, you can also put the "ContainerOverrides": { "Command": ["Ref::param_1","Ref::param_2"] } in the state machine definition instead of the in the Batch Job Definition.

ArunTnp Over a year ago

This works for me.

bluecollarcoder Jan 15 at 17:04

Just an add on: with Jsonata, the "Parameters" object is now: "Parameters": { "param_1": "{% $states.input.param_1 %}", "param_2": "{% $states.input.param_2 %}2" }

Collectives™ on Stack Overflow

AWS Step Function - passing Input to container override for AWS batch job

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related