Zach
12/08/2020, 6:39 PMJim Crist-Harif
12/08/2020, 7:00 PMDaskExecutor
with workers running in individual containers, and use resource labels to lock tasks to run in a specific worker. This is a lot more complicated setup (a lot more complicated), but could also be useful in some scenarios.
If possible, I recommend the first option. We could add a new task to the task library to create, run, and wait on a single docker container that would make this pattern easier, but it would still be for executing a CLI command inside a container not running a python function.Zach
12/08/2020, 7:17 PMJim Crist-Harif
12/08/2020, 7:27 PMIt would also be helpful to have a way to copy a file into the docker containerThat seems like a bit of scope creep, I think for that I'd want the user to write their own custom thing - prefect works fine with user-written tasks, only common patterns make sense for inclusion in the task library.
Zach
12/08/2020, 7:37 PMJim Crist-Harif
12/08/2020, 7:42 PMZach
12/08/2020, 10:34 PMJim Crist-Harif
12/08/2020, 11:02 PM