Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Good morning from Chicago!

I'd like to get confirmation on something I think I've discovered. Is this true?

&gt; If you use `KubernetesJobEnvironment` as the execution environment for a flow, any `executor` you pass will be ignored and only the default executor will be used

The `setup()` and `execute()` methods for `KubernetesJobEnvironment` are inherited from `Environment`, and seem to be empty:

<https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/base.py#L60-L75>

and the `run_flow()` method with `KubernetesJobEnvironment` is inherited from `Environment`, which says "run the flow with the default executor"

<https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/environments/execution/base.py#L89>

and it looks like the default executor is loaded from this config, and is set to `LocalExecutor`: <https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/config.toml#L141>

I'm trying this combination, and was confused why my flows seem to run successfully but don't attempt to create a Dask cluster

• storage: `Docker`
• environment: `KubernetesJobEnvironment`
• executor: `DaskExecutor`
• agent: `KubernetesAgent`
• registered with / run from Prefect Cloud

Are you running on master? Or the latest release?

These changes were just made to allow setting executors on those environments, but haven't yet been released.

Ah yeah, should have clarified! I'm running on the most recent commit to `master` as of this morning. Those links above are to the most recent commit on `master`.

Hmmm, things should run fine as configured. Do you have a reproducible example?

The docstrings might be incorrect in a few places (I might have missed a few), but the code should use the configured executor.

I can try to make one. Also maybe I got the wrong commit in my links. If you tell me "it's definitely expected that you can mix a `KubernetesJobEnvironment` and `DaskExecutor`", then I probably made a mistake along the way here.

It is expected, but there might still be bugs. If you want to try things out it'd be useful to get some external validation.

Ok sure, thanks! It looks like those links I posted above are NOT from the latest commit. Bad click on my part.

I can see on `master` that `Environment.run_flow()` (which `KubernetesJobEnvironment` inherits) is now looking at the `executor` you pass: <https://github.com/PrefectHQ/prefect/blob/2746ba22ba73540e926e2ca6080591081c412ce6/src/prefect/environments/execution/base.py#L133-L141>

ok let me try this again and check the places where I'm pegging to the latest commit. Maybe I made a mistake rebuilding the agent image. Sorry for the confusion!

it's super useful just to get confirmation that what I'm trying _should_ work, so thank you for the quick response!

No worries, thanks for trying things out! We're definitely trying to simplify the deployment configuration, so please speak up as you find rough edges.

I think I found the root cause of the behavior I was seeing!

<https://github.com/PrefectHQ/prefect/pull/2849>