Hello all, I have been looking for if prefect 2 is able to leverage different container for different tasks. At Datateer we are leveraging kubenetes as our deployment model. There was a post made on the
discord asking the same question. I also found on the
discord for prefect 1 that there are ideas that lead me to the
kubernetes job task. This lead me to find this
prefect-kubernetes package which provides support for prefect 2. I also know that there is the native ability to run sub-flows in a task runner through
dask.
My question is if there is an accepted standard for managing tasks that one would want to run in an isolated container to better manage dependency conflicts and design runtimes for the tasks that are optimized for the needed resources. The latter would help with reducing runtime costs.