https://prefect.io logo
Title
h

haris khan

06/29/2022, 6:31 PM
can we do resource managment in prefect ? giving different flows differnt cpu power
1
b

Billy McMonagle

06/29/2022, 8:57 PM
You can do this with
KubernetesRun
, not sure about other configurations. For example:
KubernetesRun(
        env=environment_variables,
        image=f"{IMAGE_REPOSITORY}:{flow_image}-{GIT_SHA}",
        image_pull_policy="IfNotPresent",
        service_account_name=service_account_name,
        job_template=job_template,
        cpu_request=cpu_request,
        memory_request=memory_request,
    )
h

haris khan

06/29/2022, 9:59 PM
i know this can be done in prefect 1.0 but can we do resource managment in prefect 2.0? and how?
1
k

Kevin Kho

06/30/2022, 8:39 AM
It would be through the FlowRunner. What FlowRunner are you using?
h

haris khan

06/30/2022, 9:35 AM
kubernetes flowrunner
a

Anna Geller

06/30/2022, 11:50 AM
Prefect can orchestrate and manage your data flows, but it's not responsible for resource management - this is what Kubernetes or cloud provider services are for - if you are familiar with Kubernetes, this would be the easiest and most straightforward way to solve that problem but we are totally aware that it's still not straightforward to do and we are working on making this process easier later this year