https://prefect.io logo
Title
j

Jonathan Langlois

03/13/2023, 2:08 AM
Hi everyone šŸ‘‹ Is there any plan to support infrastructure configuration at runtime with flow parameters and not only at deployment time with deployment specs or at initialization time with task runner configuration? Currently, we have to define the flow pod k8s resources when we deploy it and dask task runner specs before the flow is executed, but having a way to configure both with flow parameters would be convenient!
šŸ‘€ 2
t

Toby Rahloff

03/13/2023, 9:44 AM
Hi @Jonathan Langlois, we ran into the same issue a while ago and there is an ongoing discussion with @Chris White over at Discourse if you are interested šŸ™Œ Would love to hear your perspective and whether the options discussed in this thread would also work for your scenario. https://discourse.prefect.io/t/dynamically-set-flow-configuration-via-flow-run-parameters/2301/7
:upvote: 1
j

Jonathan Langlois

03/13/2023, 11:42 PM
Hi @Toby Rahloff and @Chris White. Thank you for the discourse link šŸ™ It seems that the upcoming feature could solve our use case regarding the
task_runner
configuration at runtime. If I understand properly, it will look like this:
from prefect import runtime # namespace for accessing dynamic runtime info as-needed


def get_task_runner():
    task_runner_specs = runtime.deployment.parameters.get("task_runner_specs", "default specs here")
    return DummyTaskRunner(task_runner_specs=task_runner_specs)


@flow(task_runner=get_task_runner())
def dummy_flow(
    task_runner_specs, # <-- Not sure if we'll have to define this flow argument there
    dummy_arg,
)
    # Enjoy the custom task runner configuration by flow run
If this accurate, it would be really convenient! However, this won't solve the other use case: having a way to override the flow infrastructure (defined at deployment time) when running a flow. For now, we have several deployments with different configuration for the size of the k8s pod that will run the flow. We would like to let our users decide the size of this pod as well. But the task_runner override at runtime is the priority for us. Glad it will be supported soon šŸŽ‰
c

Chris White

03/13/2023, 11:45 PM
for the task runner, that's exactly right! i'm working on this feature right now so hope to get it in the release this week. For the infrastructure configuration, we have two paths forward: • we will be rolling out some upgrades to the Work Pool concept that will let you publish your own infrastructure configuration on the work pool (so for example, you would choose to expose the pod size as an option); individual deployments could then provide values for these pieces of configuration. Note that this is still similar to the current setup where each deployment still has a static setting. • we could consider exposing an API for letting individual flow runs also override these same values and take precedence over the deployment-specific overrides. I think this makes sense as a feature enhancement for sure.
🄳 1
šŸ‘ 1
j

Jonathan Langlois

03/13/2023, 11:55 PM
Thanks for the quick reply. Looks promising. Looking forward to start using it!
šŸš€ 2
t

Toby Rahloff

03/14/2023, 3:58 PM
• we could consider exposing an API for letting individual flow runs also override these same values and take precedence over the deployment-specific overrides. I think this makes sense as a feature enhancement for sure.
That sounds extremely convenient @Chris White! This would also be a feature that addresses the challenge outlined in the Discourse thread. Do you have any gut feeling on the rough ETA of this?
c

Chris White

03/14/2023, 4:07 PM
hmmm i might be able to get that work wrapped into our current worker sprint; i don't want to promise that though so let me check with the team and get back to you shortly
šŸ™Œ 1
t

Toby Rahloff

03/14/2023, 4:16 PM
#8790 is amazing work šŸ‘ Already made many engineers in the LiveEO Slack happy and excited this morning 😁
šŸ™Œ 1
Thanks a lot for pushing this topic!
šŸ’Æ 2
c

Chris White

03/14/2023, 4:24 PM
anytime, thanks for the interesting use case!
šŸ™Œ 1