Jonathan Langlois
03/13/2023, 2:08 AMToby Rahloff
03/13/2023, 9:44 AMJonathan Langlois
03/13/2023, 11:42 PMtask_runner
configuration at runtime. If I understand properly, it will look like this:
from prefect import runtime # namespace for accessing dynamic runtime info as-needed
def get_task_runner():
task_runner_specs = runtime.deployment.parameters.get("task_runner_specs", "default specs here")
return DummyTaskRunner(task_runner_specs=task_runner_specs)
@flow(task_runner=get_task_runner())
def dummy_flow(
task_runner_specs, # <-- Not sure if we'll have to define this flow argument there
dummy_arg,
)
# Enjoy the custom task runner configuration by flow run
If this accurate, it would be really convenient!
However, this won't solve the other use case: having a way to override the flow infrastructure (defined at deployment time) when running a flow. For now, we have several deployments with different configuration for the size of the k8s pod that will run the flow. We would like to let our users decide the size of this pod as well.
But the task_runner override at runtime is the priority for us. Glad it will be supported soon šChris White
03/13/2023, 11:45 PMJonathan Langlois
03/13/2023, 11:55 PMToby Rahloff
03/14/2023, 3:58 PM⢠we could consider exposing an API for letting individual flow runs also override these same values and take precedence over the deployment-specific overrides. I think this makes sense as a feature enhancement for sure.That sounds extremely convenient @Chris White! This would also be a feature that addresses the challenge outlined in the Discourse thread. Do you have any gut feeling on the rough ETA of this?
Chris White
03/14/2023, 4:07 PMToby Rahloff
03/14/2023, 4:16 PMChris White
03/14/2023, 4:24 PM