flavienbwk

    flavienbwk

    1 year ago
    Hi, I have a question about Prefect agents and Flows : is there something in Prefect such as Kubernetes'
    resources
    parameters to specify resource affinities (RAM, CPU cores, GPU environment) a Flow requires to be run ? Ex (Kubernetes) :
    resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
    Thanks !
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @flavienbwk, not super sure if this is what you need but the RunConfig takes in resource requests like
    flow.run_config = KubernetesRun(cpu_request=2, memory_request="2Gi")
    as seen here . If you need affinity, for example you need a flow to run on a specific node, you can use nodeAffinity in the job template. Does that answer you?
    flavienbwk

    flavienbwk

    1 year ago
    Thank you for this option ! I would like to use exclusively Prefect and its agents instead of Kubernetes jobs. Does Prefect support auto-detection of its agents resources (CPU cores count, RAM count) and is able to support the same parameters as for
    KubernetesRun
    ?
    Kevin Kho

    Kevin Kho

    1 year ago
    Auto-detection I think is not supported, but maybe you can try setting it high and see what happens? I guess the pod startup might fail if it can’t get the resources?
    flavienbwk

    flavienbwk

    1 year ago
    I mean I would like to avoid using Kubernetes. And exclusively use VM-runned Prefect agents.
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah I see if you use the LocalAgent or DockerAgent, it just keeps starting flows without any knowledge of the resources or utilization.
    flavienbwk

    flavienbwk

    1 year ago
    Thank you for these precisions ! Might be an idea for future Prefect features 🙂
    Kevin Kho

    Kevin Kho

    1 year ago
    What would you want the behavior to be? Since the LocalAgent is already designed to use as much resources as possbiel?
    flavienbwk

    flavienbwk

    1 year ago
    My configuration is the following : I have 5 VPS with different capabilities (some with lot of RAM, some with low RAM, some high CPU cores, low CPU cores). Each VPS has 1 Prefect agent instanciated. I would like Prefect to automatically be able to distribute the flow I've registered with minimum requirements as parameter (
    (cpu_request=2, memory_request="2Gi")
    ) to an agent with sufficient resources.
    Kevin Kho

    Kevin Kho

    1 year ago
    Gotcha, I think the prerequisite of that is that it’s possible to set limits with Python on a process that is started, which I’m not sure it is for all OSs. Will take a look a it.
    flavienbwk

    flavienbwk

    1 year ago
    Let me refine my thinking : • If Prefect API distributes tasks to agents : Prefect API should know which agent has which resources (RAM/CPU) to send the Flow to the agent with enough Flow's requested resources. • If agents listen to Prefect API tasks : Agent should make sure it is able to accept the Flow depending on its resources and Flow's requested resources.
    Kevin Kho

    Kevin Kho

    1 year ago
    I understand the intention but to be honest, this can be managed by a user with labels, so I don’t know when a feature like this would be picked up. Also, Cloud has the task run concurrency and flow run concurrency which queue flows/tasks if too many with the same label/tag are already running. This is the first time I’ve seen this request. I’ll keep an eye out if I see it pop up more.
    flavienbwk

    flavienbwk

    1 year ago
    Yes labels are really useful. For example with agents having GPU capabilities (label e.g:
    cuda
    ). And this is what I'll be using for the moment while I don't have Kubernetes. BUT, for quantities-related characteristics (such as RAM amount or CPU count), it can't really be used.
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! Just to add some additional context, resource based scheduling is considered out of scope for Prefect. Scheduling jobs to consume resources optimally is a very complex problem that is better addressed by dedicated tooling (like Kubernetes).
    flavienbwk

    flavienbwk

    1 year ago
    Hi @Michael Adkins thank you for these addings. My problem could be addressed just by implementing "countable tags" like cpu-xx or ram-xx or vram-xx. It might just be tags provided by the user (declarative values in agent's config). These countable tags would make it possible to perform comparison operations on them (greater than, equal, lower than...).
    Michael Adkins

    Michael Adkins

    1 year ago
    As Kevin has noted we do support this via concurrency limiting by tag.
    flavienbwk

    flavienbwk

    1 year ago
    Do you have any link in the documentation that points to this? Thanks
    flavienbwk

    flavienbwk

    1 year ago
    Yeah OK it's a Cloud feature. Thanks for the link