https://prefect.io
Join Slack
Hi prefect team, I am wondering if there is a plan to implement executor callbacks now that 0.14.0 ...
# ask-community
m
Hi prefect team, I am wondering if there is a plan to implement executor callbacks now that 0.14.0 has migrated away from environments. I see this discussed here https://prefect-community.slack.com/archives/CL09KU1K7/p1602269846235100 but I don't see any mention of this in the changelog.
My specific use case is to dynamically configure the cluster based on flow parameters - my current planned workaround is to make use of the 
prefect.context
in the 
cluster_class
callable and the modify the context when running the flow (not ideal but I can't think of another approach) EDIT - after some thought - modifying the context dynamically in the flow run to modify the adapt kwargs perhaps isn't as bad as I first thought (will have to wrap the flow with an outer flow basically)... It would be nice to get a confirmation if this is the best practice right now...
z
m
Hi @Zanie - thanks but this is about dynamically modifying the run config not the executor (DaskExecutor in my case)
z
Ah my apologies!
👍 1
@Jim Crist-Harif it looks like you got this pretty fleshed out in the thread they linked, not sure where this is on your current roadmap.
j
Hi Marwan, using a custom function in 
cluster_class
is the current recommended solution. This should give you access to everything in 
prefect.context
to create your cluster. What info do you need to dynamically make decisions here?
m
@Jim Crist-Harif thank you for your response - I have a number of splits parameter that based on which I need to set the minimum to maximum workers range
so it seems to me I will require a flow of flows where the outer flow defines the context given the parameter value (specifically perfect.context.min_workers and prefect.context.max_workers) to be used in the actual inner flow
or are parameters directly accessible in the context ?
z
prefect.context.parameters
m
ah ok great then
thank you @Zanie @Jim Crist-Harif
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by