Thinking of using Coiled for our Prefect stuff - how does it work to have different Docker images for different flows with Coiled? Can you just set the Execution environment as the Dask cluster and the Run_Config as DockerRun with the image file?

Hey @Matthew Alhonte, think of it as two different machines, and the machine with the Prefect agent is sending work to the Coiled cluster. In that sense, the configurations will be separated and you need the dependencies installed in both places. You can create a Coiled

software_environment

from a container. The software environment would be installed on the Dask cluster.

Thanks for the quick reply! Hrm, so the machine the Agents run on would have to get a little beefier? Right now I'm using a Micro server to run the agents and kick off Google Cloud Run jobs - I think I'd need a bigger one if it'd need to have all the same packages?

I think it can run Docker, but the image might be another thing (if you have big depdendencies). You can use a local agent and just connect to the Coiled executor that way? You will get errors if your imports are at the top of the script, but you can defer imports by moving them into the tasks, and that might work. They will be executed on the Dask cluster when the tasks run

Ah, so I think I'm seeing it - so it'd be LocalRun, and then I make a Coiled Software Environment from the Dockerfile, then I spin up the Cluster with that Software Environment? (as long as the imports are in a Task?)

Yes exactly, then you just need Python and Prefect on the micro to offload the work to the executor and run the agent.

Sweet, thanks!

I think you just need to make sure Prefect versions on the agent and flow are consistent otherwise you may get weird errors.

Kk! Yeah I'll try n make a habit of upgrading the version on the server and the one pinned in the Dockerfile at the same time.

Yeah it’s very painful to always build that software environment. It’s very helpful to decouple it from the Flow code if possible so that flow code changes don’t cause re-builds. The CI/CD also helps versioning. But of course, I recommend that POC first of LocalAgent + CoiledExecutor just to see everything works properly.