https://prefect.io logo
r

ravi

09/14/2023, 3:40 AM
Hi Team, Appreciate your help with this. How to effectively trigger the flow execution (create a flow run) when file arrives to S3 Bucket? I am thinking to leverage the event from S3 which then triggers lambda function and aws lambda function then calls the client.create_flow_run_from_deployment? Is that the right approach? Should I be able to call the create_flow_run_from_deployment from lambda given the limitation on library size? Or I should be looking into Rest API? Are there any other-ways to start the flow I should be exploring? Could you please point to some samples?
r

redsquare

09/14/2023, 6:18 AM
could use prefect webhooks from sqs https://docs.prefect.io/latest/guides/webhooks/
t

Tom Klein

09/14/2023, 9:34 AM
if you’re already in the context of a lambda function you don’t need webhooks imho since you can control 100% of the behavior, so just make a REST API call to invoke the deployment. The disadvantage of this is that it creates a tight coupling between the lambda and the delpoyment. the disadvantage of webhooks is that they pass through the public internet with no inherent security. there is also a conceptual alternative (if this type of scenario is a common pattern and you’d like to have a lot of different triggers like these, or - you prefer to work in batch (and not react to every single file arrival immediately) - where instead of triggering the deployment directly, you simply push these events to some kind of queue (SQS, kafka, or anything else) and then consume the queue in a periodically/continuously running flow
r

redsquare

09/15/2023, 4:09 AM