Pass data across loop in AWS Step Functions

Question

I have an AWS Step Function that consists of a "Map" (looping over an Array in the input), and then a number of tasks carried out in sequence with all the results. However, in the last step I'd also like to be able to access some of the original input data, that was not used in the intermediate steps. How should I best do this?

What I have tried so far:

Passing all input data through the map tasks. This however fails quickly, because there is a limit on how large results can be passed between functions, and with hundreds of iteration the original input is multiplied hundreds of times.
Using Parallel state, and add an empty state in one branch of the parallel execution that simply passes the input on, while all of the iteration happens in the other. This feels like a hack, because my step structure now poorly reflects the actual logic of the code.

What would be the “right” way to do this?

                     Iterate
                     over Fiz
                     +--------+   +-------------+    +------------+
{                    | +---+  |   |             |    |            |
  Foo: "Bar", --> -------+ |  +---+      Y      +----+     Z      |
  Fiz: [...],        | +-+ |      |             |    |            |
}                    +-----+      +-------------+    +------------+

In box Z above, I want to have access to both 'Foo', and the output of Y

Yves M. · Accepted Answer · 2023-12-13 17:31:32Z

3

It turns out that each step in the SFN has access to the original input, but you have to explicitly state insert them using Parameters. All the input is available in $$.Execution.Input.

When using Parameters, you are overriding all other input, so you will also have to define what you want to pass along from the previous step. In other words: If Y is passing {fruit: "apple"} to Z, and you want Z to see both that and the original input {Foo: "Bar"} , you would add the following to the Z step in SFN:

{
    "Parameters": {
        "Foo.$": "$$.Execution.Input.Foo",
        "fruit.$": "$.apple"
}

Note the double $$ in the first case, and the single $ in the second.

edited Dec 13, 2023 at 17:31

Yves M.

31.3k24 gold badges111 silver badges154 bronze badges

answered Feb 6, 2020 at 13:13

leo

8,6617 gold badges56 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pass data across loop in AWS Step Functions

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related