https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
j

james.lamb

06/23/2020, 3:24 PM
Good morning from Chicago! I'd like to get confirmation on something I think I've discovered. Is this true?
If you use 
KubernetesJobEnvironment
as the execution environment for a flow, any 
executor
you pass will be ignored and only the default executor will be used
The 
setup()
and 
execute()
methods for 
KubernetesJobEnvironment
are inherited from 
Environment
, and seem to be empty: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/base.py#L60-L75 and the 
run_flow()
method with 
KubernetesJobEnvironment
is inherited from 
Environment
, which says "run the flow with the default executor" https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/environments/execution/base.py#L89
and it looks like the default executor is loaded from this config, and is set to `LocalExecutor`: https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/config.toml#L141
I'm trying this combination, and was confused why my flows seem to run successfully but don't attempt to create a Dask cluster • storage: 
Docker
• environment: 
KubernetesJobEnvironment
• executor: 
DaskExecutor
• agent: 
KubernetesAgent
• registered with / run from Prefect Cloud
j

Jim Crist-Harif

06/23/2020, 3:39 PM
Are you running on master? Or the latest release?
These changes were just made to allow setting executors on those environments, but haven't yet been released.
j

james.lamb

06/23/2020, 3:40 PM
Ah yeah, should have clarified! I'm running on the most recent commit to 
master
as of this morning. Those links above are to the most recent commit on 
master
.
j

Jim Crist-Harif

06/23/2020, 3:41 PM
Hmmm, things should run fine as configured. Do you have a reproducible example?
The docstrings might be incorrect in a few places (I might have missed a few), but the code should use the configured executor.
j

james.lamb

06/23/2020, 3:42 PM
I can try to make one. Also maybe I got the wrong commit in my links. If you tell me "it's definitely expected that you can mix a 
KubernetesJobEnvironment
and `DaskExecutor`", then I probably made a mistake along the way here.
j

Jim Crist-Harif

06/23/2020, 3:44 PM
It is expected, but there might still be bugs. If you want to try things out it'd be useful to get some external validation.
j

james.lamb

06/23/2020, 3:45 PM
Ok sure, thanks! It looks like those links I posted above are NOT from the latest commit. Bad click on my part. I can see on 
master
that 
Environment.run_flow()
(which 
KubernetesJobEnvironment
inherits) is now looking at the 
executor
you pass: https://github.com/PrefectHQ/prefect/blob/2746ba22ba73540e926e2ca6080591081c412ce6/src/prefect/environments/execution/base.py#L133-L141
ok let me try this again and check the places where I'm pegging to the latest commit. Maybe I made a mistake rebuilding the agent image. Sorry for the confusion!
it's super useful just to get confirmation that what I'm trying should work, so thank you for the quick response!
j

Jim Crist-Harif

06/23/2020, 3:47 PM
No worries, thanks for trying things out! We're definitely trying to simplify the deployment configuration, so please speak up as you find rough edges.
j

james.lamb

06/24/2020, 6:16 AM
I think I found the root cause of the behavior I was seeing! https://github.com/PrefectHQ/prefect/pull/2849
Powered by