https://prefect.io logo
Title
j

james.lamb

06/23/2020, 3:24 PM
Good morning from Chicago! I'd like to get confirmation on something I think I've discovered. Is this true?
If you use
KubernetesJobEnvironment
as the execution environment for a flow, any
executor
you pass will be ignored and only the default executor will be used
The
setup()
and
execute()
methods for
KubernetesJobEnvironment
are inherited from
Environment
, and seem to be empty: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/base.py#L60-L75 and the
run_flow()
method with
KubernetesJobEnvironment
is inherited from
Environment
, which says "run the flow with the default executor" https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/environments/execution/base.py#L89
and it looks like the default executor is loaded from this config, and is set to `LocalExecutor`: https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/config.toml#L141
I'm trying this combination, and was confused why my flows seem to run successfully but don't attempt to create a Dask cluster • storage:
Docker
• environment:
KubernetesJobEnvironment
• executor:
DaskExecutor
• agent:
KubernetesAgent
• registered with / run from Prefect Cloud
j

Jim Crist-Harif

06/23/2020, 3:39 PM
Are you running on master? Or the latest release?
These changes were just made to allow setting executors on those environments, but haven't yet been released.
j

james.lamb

06/23/2020, 3:40 PM
Ah yeah, should have clarified! I'm running on the most recent commit to
master
as of this morning. Those links above are to the most recent commit on
master
.
j

Jim Crist-Harif

06/23/2020, 3:41 PM
Hmmm, things should run fine as configured. Do you have a reproducible example?
The docstrings might be incorrect in a few places (I might have missed a few), but the code should use the configured executor.
j

james.lamb

06/23/2020, 3:42 PM
I can try to make one. Also maybe I got the wrong commit in my links. If you tell me "it's definitely expected that you can mix a
KubernetesJobEnvironment
and `DaskExecutor`", then I probably made a mistake along the way here.
j

Jim Crist-Harif

06/23/2020, 3:44 PM
It is expected, but there might still be bugs. If you want to try things out it'd be useful to get some external validation.
j

james.lamb

06/23/2020, 3:45 PM
Ok sure, thanks! It looks like those links I posted above are NOT from the latest commit. Bad click on my part. I can see on
master
that
Environment.run_flow()
(which
KubernetesJobEnvironment
inherits) is now looking at the
executor
you pass: https://github.com/PrefectHQ/prefect/blob/2746ba22ba73540e926e2ca6080591081c412ce6/src/prefect/environments/execution/base.py#L133-L141
ok let me try this again and check the places where I'm pegging to the latest commit. Maybe I made a mistake rebuilding the agent image. Sorry for the confusion!
it's super useful just to get confirmation that what I'm trying should work, so thank you for the quick response!
j

Jim Crist-Harif

06/23/2020, 3:47 PM
No worries, thanks for trying things out! We're definitely trying to simplify the deployment configuration, so please speak up as you find rough edges.
j

james.lamb

06/24/2020, 6:16 AM
I think I found the root cause of the behavior I was seeing! https://github.com/PrefectHQ/prefect/pull/2849