@Marvin A rather fundamental question; I have prefect worker and pool, and server set up on a kubernetes cluster. I am looking to run a flow of tasks where each task is ideally a job requiring running a container with some input params. Images related to tasks all exist on a private ecr repo. What is the best strategy? Prior to this I was thinking to have a flow doing

map

on the task so I can run them concurrently. But I probably can not have a task that complicated. The way I understand the above tasks now, it seems each of them should be a flow. So now I have a basic question, how do I run a container from prefect so it does some stuff. Im not looking to run my flow inside container. It seems thats what deploying a flow with

job_variables

and running it does.

Based on the information from the Prefect documentation, here are some key points: 1. Prefect workers bridge the Prefect orchestration environment with your execution environment. When a deployment creates a flow run, it is submitted to a specific work pool for scheduling. A worker running in the execution environment can poll its respective work pool for new runs to execute, or the work pool can submit flow runs to serverless infrastructure directly, depending on your configuration. More Info 2. Each deployment is associated with a single flow, but any given flow can be referenced by multiple deployments. Deployments are uniquely identified by the combination of flow_name/deployment_name. More Info 3. You can verify your Prefect worker deployment on Kubernetes by running

kubectl get pods -n prefect

. You should see the Prefect worker pod running. More Info As for running a container from Prefect, I couldn't find a direct answer in the documentation. However, since Prefect is a workflow orchestration tool, it should be possible to define a task that runs a container. You might need to use the Kubernetes API or a Kubernetes Python client to create and manage the container from within the task. As for using

map

for concurrent execution, it should be possible as long as each task is independent and can be run concurrently. Please note that this is a general guidance and you might need to adjust based on your specific use case and environment.