https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Andy Dang

07/15/2022, 5:10 PM
Hi! I just started to look into Prefect as an alternative to AWS Step Functions. We use Step Functions to orchestrate our Databricks jobs with AWS Lambda - and Prefect supports Databricks (so the main actual logic is in DB - we just need Prefect to orchestrate tasks) However what I can’t figure out is where to run the prefect agent to manage my Databricks jobs? Do I spin up an EC2 instance and kick it off there? Can I run it as a Docker container in my ECS cluster? Doubt that I can run it as a Lambda (though that would be amazing), but it doesn’t look like there’s an easy way to do it. Any recommendations?
1
a

Anna Geller

07/15/2022, 5:12 PM
I believe this may be what you need - check master branch here https://github.com/anna-geller/prefect-dataops this is WIP, I'll write a post documenting that in the next weeks
you should be able to deploy the agent with a single click
a

Andy Dang

07/15/2022, 5:47 PM
Thanks. Where are these env variables like PREFECT_API_KEY documented? I can’t find them in the doc site
a

Anna Geller

07/15/2022, 6:40 PM
Follow the orion-docs.prefect.io to see how to create the API key
5 Views