Hi Team, Appreciate your help with this. How to e...
# ask-community
r
Hi Team, Appreciate your help with this. How to effectively trigger the flow execution (create a flow run) when file arrives to S3 Bucket? I am thinking to leverage the event from S3 which then triggers lambda function and aws lambda function then calls the client.create_flow_run_from_deployment? Is that the right approach? Should I be able to call the create_flow_run_from_deployment from lambda given the limitation on library size? Or I should be looking into Rest API? Are there any other-ways to start the flow I should be exploring? Could you please point to some samples?
r
could use prefect webhooks from sqs https://docs.prefect.io/latest/guides/webhooks/
t
if you’re already in the context of a lambda function you don’t need webhooks imho since you can control 100% of the behavior, so just make a REST API call to invoke the deployment. The disadvantage of this is that it creates a tight coupling between the lambda and the delpoyment. the disadvantage of webhooks is that they pass through the public internet with no inherent security. there is also a conceptual alternative (if this type of scenario is a common pattern and you’d like to have a lot of different triggers like these, or - you prefer to work in batch (and not react to every single file arrival immediately) - where instead of triggering the deployment directly, you simply push these events to some kind of queue (SQS, kafka, or anything else) and then consume the queue in a periodically/continuously running flow
r
Thanks @Tom Klein @redsquare for thoughts. Do you think is it possible to do something like S3 File-> AWS EventBridge -> calling web hooks (without needing then lambda)? Is it true that WebHoocks can accept Prefect pay load which can trigger the flow? @Tom Klein I saw an python client.create_flow_run_from_deployment do you think this could be useful to call from lambda?
r