AWS Step Function, Map State and Batch Input/Output

Question

Let's suppose I have a Step Function with a Map State. The Map State is a Batch Job, associated with a Docker container. I want pass input parameters to containers, and receive output for other SF's states.

I believe it could be a Lambda Function, iterating thru the input as array, and pass each element as environment variables set to containers. But how could the lambda working with foreach + environment variables look like? How can I catch Docker container output (I believe it could be S3 file/directory)?

Also is there any alternative to a Lambda Function at all?

Bwyss · Accepted Answer · 2021-08-27 16:25:30Z

1

Handling the iterator:

If you have a predefined input array that you want to iterate on with the map state, then you can just pass that as the Map InputPath and ItemsPath, but in some cases you may need to setup a lambda that that will go and create that list for you.

Your ItemsPath might looks something like:

 "list": [
          {
            "input": "<my_cool_input parameters>"
          },
          {
            "input": "<my_cool_input parameters>"
          }...
         ]

Handling the output:

As far as I know currently there is no way to get an output from batch compute back to the state machine directly. So you will need to take an indirect approach.

One way could be to write the output from your docker container to some temporary location such as dynamodb or s3. Then you would need a step in your step function to read the output from dynamodb (you can do that directly, no lambda needed, if you write to s3 then you will need a lambda to read the output).

It would seem that this approach would also be needed to capture raised exceptions from a docker container - I'm all ears if anyone has a better approach.

answered Aug 27, 2021 at 16:25

Bwyss

1,8363 gold badges28 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

anton Over a year ago

Seems it is the best solution for the moment. Thanks a lot!

Collectives™ on Stack Overflow

AWS Step Function, Map State and Batch Input/Output

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related