3

I have a step function that runs a series of lambda functions. I would like to set up a lambda that will trigger this each day when ALL of the 3 input files have been updated.

I know you can trigger off of s3 events, but how can I have the requirement that all three have to have been updated? That is, I don't want the trigger to fire on any of the three events ('or' condition), I want it to trigger when all three have been updated ('and' condition).

How would I approach setting up such a trigger event?

Thanks

Improve this question
3
  • Can I continue on the same note? Don't know why would you "private" the question and ask it again. Commented Aug 21, 2020 at 16:23
  • 1
    Sorry, as I explained in the comment I think the question was a bit misdirected. It's a question about lambda triggers, not step functions. This should be clearer for everyone Commented Aug 21, 2020 at 17:00
  • 1
    No worries, I have posted my answer here too, you can check on that. Commented Aug 22, 2020 at 2:38

3 Answers 3

Reset to default
2

There is no way to define a trigger that says "only run after these 3 files have been uploaded".

You would need to implement this logic yourself.

The uploads would trigger an AWS Lambda function. This function should check for the existence of the other two files. If they are present, it should continue processing. If they are not present, it should exit.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

As if now, there is no direct way to tigger Lambda on the basis of such conditions. You need to write your logic in a separate lambda which will trigger your step function.

You can refer here about how to trigger a Step Function from Lambda.

Now, coming to the point about logic in your Lambda which will check for your files. Here, I believe you know all those three files or at-least you know the file prefix so when ever any file is uploaded to S3 you can check in Lambda if three files are there or not. You can trigger the state machine directly from Lambda if all three files are found else you can stop there itself. Once the Step Machines execution is complete either you can clean up those three files or you can rename it to something else or you can move those files to a different "folder" so that next time when you upload any of those three files your lambda does not triggers state machine.

The other logic I could think of is, uploading all these files together as zip. Once the zip is uploaded to S3, your lambda gets notified, processes that zip and extracts all three files, does some processing and triggers the state machine with processed data.

Improve this answer

Comments

1

Here is the solution I ended using. I'm using a single lambda with a trigger for each file. For each trigger, the lambda adds the name of the file that triggered it to a file stored on s3. It then checks if the file has all the required file names to continue. If it does, it kicks off the stepfunction and clears out the list. I am capturing the name of the file that triggers the lambda from the event object that is passed in by default via lambda_function(event,context).

So on a given day, the same lambda gets triggered multiple times, recording the filename each time until it has collected them all.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.