https://prefect.io logo
m

Marwan Sarieddine

12/28/2020, 2:31 PM
Hi prefect team, I am wondering if there is a plan to implement executor callbacks now that 0.14.0 has migrated away from environments. I see this discussed here https://prefect-community.slack.com/archives/CL09KU1K7/p1602269846235100 but I don't see any mention of this in the changelog.
My specific use case is to dynamically configure the cluster based on flow parameters - my current planned workaround is to make use of the
prefect.context
in the
cluster_class
callable and the modify the context when running the flow (not ideal but I can't think of another approach) EDIT - after some thought - modifying the context dynamically in the flow run to modify the adapt kwargs perhaps isn't as bad as I first thought (will have to wrap the flow with an outer flow basically)... It would be nice to get a confirmation if this is the best practice right now...
z

Zanie

12/28/2020, 3:29 PM
m

Marwan Sarieddine

12/28/2020, 3:37 PM
Hi @Zanie - thanks but this is about dynamically modifying the run config not the executor (DaskExecutor in my case)
z

Zanie

12/28/2020, 3:49 PM
Ah my apologies!
👍 1
@Jim Crist-Harif it looks like you got this pretty fleshed out in the thread they linked, not sure where this is on your current roadmap.
j

Jim Crist-Harif

12/28/2020, 8:11 PM
Hi Marwan, using a custom function in
cluster_class
is the current recommended solution. This should give you access to everything in
prefect.context
to create your cluster. What info do you need to dynamically make decisions here?
m

Marwan Sarieddine

12/29/2020, 12:21 AM
@Jim Crist-Harif thank you for your response - I have a number of splits parameter that based on which I need to set the minimum to maximum workers range
so it seems to me I will require a flow of flows where the outer flow defines the context given the parameter value (specifically perfect.context.min_workers and prefect.context.max_workers) to be used in the actual inner flow
or are parameters directly accessible in the context ?
z

Zanie

12/29/2020, 12:23 AM
prefect.context.parameters
m

Marwan Sarieddine

12/29/2020, 12:23 AM
ah ok great then
thank you @Zanie @Jim Crist-Harif