https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Matthew Ferry

12/05/2022, 3:06 PM
Hello! I'm working with prefect to orchestrate some dbt tasks, but I'm struggling with ECS deployment. I have one flow that I want to run for multiple customers, so I need data access to be single-tenant, but it doesn't seem efficient to run an agent for each customer. I was hoping to: • Run one agent as a long-running ECS task that listens to one queue • When a flow run is requested, that agent spins up a separate ECS task (with client-specific permissions) to run the entire flow • The ECS task runs a prefect flow that includes dbt shell command prefect tasks from prefect-dbt Could somebody help me think through how this can be done? Most of the ECS docs describe v1 not v2, and I only found this discussion in Discourse which said ECS wasn't really supported.
I might be answering my own question but @Ryan Peden - would the workflow you described here support what I'm trying to do? https://prefect-community.slack.com/archives/CL09KU1K7/p1669664242284559?thread_ts=1669633983.961299&cid=CL09KU1K7
r

Ryan Peden

12/05/2022, 3:13 PM
Yes, I think that is essentially what you want to do here. You'd probably want to use a separate deployment for each customer so it can set the customer-specific permissions for each flow run.
1
m

Matthew Ferry

12/05/2022, 3:30 PM
That makes sense! Very helpful, thank you! For the ECS task Docker image, I have prefect installed and I have a
flow.py
file on it, but do I just give it the command
python flow.py
in the Dockerfile or should the agent supply a command? I'm not clear on how the agent and the task interact, do you have docs you recommend for that?
I managed to figure this out - the Docker image needs to run
python -m prefect.engine
and have an environment variable for the prefect API URL. Once the ECS Task has been successfully kicked off, the API just waits to be contacted and keeps the flow in a "pending" state.