Thinking of using Coiled for our Prefect stuff - h...
# ask-communitym
Matthew Alhonte
07/19/2021, 8:43 PMThinking of using Coiled for our Prefect stuff - how does it work to have different Docker images for different flows with Coiled? Can you just set the Execution environment as the Dask cluster and the Run_Config as DockerRun with the image file?
k
Kevin Kho
07/19/2021, 8:47 PM
Hey @Matthew Alhonte, think of it as two different machines, and the machine with the Prefect agent is sending work to the Coiled cluster. In that sense, the configurations will be separated and you need the dependencies installed in both places. You can create a Coiled
software_environmentm
Matthew Alhonte
07/19/2021, 8:51 PMThanks for the quick reply! Hrm, so the machine the Agents run on would have to get a little beefier? Right now I'm using a Micro server to run the agents and kick off Google Cloud Run jobs - I think I'd need a bigger one if it'd need to have all the same packages?
Matthew Alhonte
07/19/2021, 8:54 PMI guess I'd just need Docker, though? Can a Micro run Docker?
k
Kevin Kho
07/19/2021, 9:02 PM
I think it can run Docker, but the image might be another thing (if you have big depdendencies). You can use a local agent and just connect to the Coiled executor that way? You will get errors if your imports are at the top of the script, but you can defer imports by moving them into the tasks, and that might work. They will be executed on the Dask cluster when the tasks run
m
Matthew Alhonte
07/19/2021, 9:07 PMAh, so I think I'm seeing it - so it'd be LocalRun, and then I make a Coiled Software Environment from the Dockerfile, then I spin up the Cluster with that Software Environment? (as long as the imports are in a Task?)
Matthew Alhonte
07/19/2021, 9:08 PM(as you can see, "never, ever thinking about servers" is pretty important to me :p)
k
Kevin Kho
07/19/2021, 9:08 PM
Yes exactly, then you just need Python and Prefect on the micro to offload the work to the executor and run the agent.
👍 1
m
Matthew Alhonte
07/19/2021, 9:09 PMSweet, thanks!
k
Kevin Kho
07/19/2021, 9:10 PM
I think you just need to make sure Prefect versions on the agent and flow are consistent otherwise you may get weird errors.
👍 1
m
Matthew Alhonte
07/19/2021, 9:18 PMKk! Yeah I'll try n make a habit of upgrading the version on the server and the one pinned in the Dockerfile at the same time.
Matthew Alhonte
07/19/2021, 9:30 PMHrm, or maybe instead of re-making the Software Environment when I spin the cluster, I could have a GitHub Action build the Software Environments #RubberDucking
Matthew Alhonte
07/19/2021, 9:31 PMThe important thing is that people can make a flow with a .py file, a requirements.txt or environment.yml, (and sometimes a Dockerfile) and don't have to think about anything else lol
k
Kevin Kho
07/19/2021, 9:32 PM
Yeah it’s very painful to always build that software environment. It’s very helpful to decouple it from the Flow code if possible so that flow code changes don’t cause re-builds. The CI/CD also helps versioning. But of course, I recommend that POC first of LocalAgent + CoiledExecutor just to see everything works properly.
👍 1
3 Views