Hi! I just started to look into Prefect as an alternative to AWS Step Functions. We use Step Functions to orchestrate our Databricks jobs with AWS Lambda - and Prefect supports Databricks (so the main actual logic is in DB - we just need Prefect to orchestrate tasks) However what I can’t figure out is where to run the prefect agent to manage my Databricks jobs? Do I spin up an EC2 instance and kick it off there? Can I run it as a Docker container in my ECS cluster? Doubt that I can run it as a Lambda (though that would be amazing), but it doesn’t look like there’s an easy way to do it. Any recommendations?

I believe this may be what you need - check master branch here https://github.com/anna-geller/prefect-dataops this is WIP, I'll write a post documenting that in the next weeks

Thanks. Where are these env variables like PREFECT_API_KEY documented? I can’t find them in the doc site

Follow the orion-docs.prefect.io to see how to create the API key