Input and Ouput to ECS task in Step function

Question

Have previously worked with lambda orchestration using AWS step function. This has been working very well. Setting the result_path of each lambda will pass along arguments to subsequent lambda.

However, I now need to run a fargate task and then pass along arguments from that fargate task to subsequent lambdas. I have created a python script that acts as the entrypoint in the container definition. Obviously in a lambda function the handler(event, context) acts as the entrypoint and by defining a return {"return_object": "hello_world"} its easy to pass a long a argument to the next state of the state machine.

In my case though, I have task definition with a container definition created from this Dockerfile:

FROM python:3.7-slim

COPY my_script.py /my_script.py
RUN ln -s /python/my_script.py /usr/bin/my_script && \
chmod +x /python/my_script.py

ENTRYPOINT ["my_script"]

Hence, I am able to invoke the state machine and it will execute my_script as intended. But how do I get the output from this python script and pass it along to another state in the state machine?

I have found some documentation on how to pass along inputs, but no example of passing along outputs.

The step function does nor know about your Python script, just the ecs task. You would have to stage any output data from your script somewhere the lambda could read it, so a file in s3, parameter store, sqs etc. — Ronan Cunningham
– Ronan Cunningham, Commented Jun 26, 2021 at 22:41
Yeah I figured, but I have not found a good way to do it yet. Any suggestions on pattern? — Frankster
– Frankster, Commented Jun 27, 2021 at 8:46
Depends on the context. SQS is a good pattern to pass data to a single lambda but in your description you mentioned multiple lambas and it’s not clear if you mean this to be simultaneously so possibly an SNS topic could suit you best as multiple lambdas could subscribe and get the same information at the same time. Seems to be more of an architectural issue where there are always trade offs that only you are best placed to understand. — Ronan Cunningham
– Ronan Cunningham, Commented Jun 27, 2021 at 20:15

Drew Bollinger · Accepted Answer · 2022-02-10 20:50:01Z

15

To get output from an ECS/Fargate task, I think you have to use the Task Token Integration instead of Run Job (Sync) which is usually recommended for Fargate tasks. You can pass the token as a container override ("TASK_TOKEN": "$$.Task.Token"). Then inside your image you need some logic like this:

client = boto3.client('stepfunctions')
client.send_task_success(
    taskToken=os.environ["TASK_TOKEN"],
    output=output
)

to pass it back.

answered Feb 10, 2022 at 20:50

Drew Bollinger

2464 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Input and Ouput to ECS task in Step function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related