https://prefect.io logo
Title
a

Aleksandr Liadov

01/18/2023, 2:40 PM
Hello, In Prefect1 I can specified asked resource with
KubernetesRun
(
cpu_request
,
memory_request
,
cpu_limit
,
memory_limit
) In Prefect2 I use
KubernetesJob
, how can I specify the same parameters?
c

Christopher Boyd

01/18/2023, 3:17 PM
HI Aleksandr, You can create a job template and add them to the spec, or modify them using customization parameters via json patch
r

Ricky Zhang

01/18/2023, 4:22 PM
@Christopher Boyd My flow contains several parallel tasks. But when I made the flow runs in k8s job infrastructure block, I see all tasks runs in sequential manner. Will changing “.sepc.parallelism” value in job template could make Prefect engine run tasks in parallel fashion? What is the execution model? One parallel task per PVM (Python VM process) inside the same pod or one parallel task per pod?
c

Christopher Boyd

01/18/2023, 4:23 PM
Each “flow” is a k8s job, which is a pod. If you’re looking for individual tasks to be spawned as jobs themselves, you’d probably want to be looking at Dask
r

Ricky Zhang

01/18/2023, 4:24 PM
So in a nutshell, there are no parallelism in k8s job for a flow with concurrent tasks. Is that correct?
c

Christopher Boyd

01/18/2023, 4:25 PM
natively, that is correct
r

Ricky Zhang

01/18/2023, 4:28 PM
Thanks. We had a hard time to figure out how to hook up an existing Dask cluster from Prefect 2. I saw an example to specify Dask task runner in the flow script. But this doesn’t sound like the rest deployment execution model. There is no infrastructure block for that. It is strange.
j

Joao Moniz

01/20/2023, 9:19 AM
@Christopher Boyd do you know if running tasks as independent pods (kubernetes-job) is something that is on the pipeline to be developed?