Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

is there a way to use `Parameters` or something analogous as input to a `KubernetesRun` to request a specific amount of resources on the fly? I'm working w/ Fargate and want to give users flexibility to run flows with differing amounts of compute

Hi <@U03G3P45692> :wave: Am I correct in assuming that you’re using Prefect 1.0? Could you tell me a bit more about your use case?

When you create a flow run you should be able to override values of your run config, including memory and cpu:
1. Manually - in the UI, navigate to the flow you want to run, then go to the “:rocket: Run” tab and then “Show advanced run configuration”
2. Programmatically - instantiate a client and use <https://docs-v1.prefect.io/api/latest/client/client.html#client-2:~:text=prefect.client.client.Client.create_flow_run|create_flow_run> passing a <https://docs-v1.prefect.io/api/latest/run_configs.html#kubernetesrun|KubernetesRun> run_config with your desired cpu_request.

I'm using Prefect 1 yes.

The use case is that some pipelines, depending on input parameters, would require more resources to run. Also, potentially a gpu.

This would change what the KubernetesRun provisions

I have the prefect agent in default namespace running on EKS backed with fargate (currently no GPU support)

for the scenario with a GPU, it's my understanding that I would need to provision an EC2 reserved node group in a separate kubernetes namespace

Also, Is it possible to use the default namespaced agent to dispatch jobs to the namespace that has GPU resources?

e.g. can I interleave flows that would run on fargate and ec2?

My goal is to be able to stitch together flows, on the fly, underneath parent flows. And the execution instance (fargate, ec2) for individual subflows underneath the parent flow may be different

I think it should be possible to selectively place flows on different clusters and/or capacity providers, but I’ll have to look into it more. Another approach would be to just run two agents, each pointing to a different k8s run config. Since the agents aren’t doing the work themselves, they require very few resources to run. Then you can use <https://docs-v1.prefect.io/orchestration/agents/overview.html#labels|labels> to direct work to the appropriate agent. 