Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
j
james.lamb
06/23/2020, 3:24 PMGood morning from Chicago!
I'd like to get confirmation on something I think I've discovered. Is this true?
If you useas the execution environment for a flow, anyKubernetesJobEnvironmentyou pass will be ignored and only the default executor will be usedexecutor
The
setup()execute()KubernetesJobEnvironmentEnvironmentrun_flow()KubernetesJobEnvironmentEnvironmentand it looks like the default executor is loaded from this config, and is set to `LocalExecutor`: https://github.com/PrefectHQ/prefect/blob/fbe0d958d98db5c10013cbcf29cebeca77ba6bc7/src/prefect/config.toml#L141
I'm trying this combination, and was confused why my flows seem to run successfully but don't attempt to create a Dask cluster
• storage:
DockerKubernetesJobEnvironmentDaskExecutorKubernetesAgentj
Jim Crist-Harif
06/23/2020, 3:39 PMAre you running on master? Or the latest release?
These changes were just made to allow setting executors on those environments, but haven't yet been released.
j
james.lamb
06/23/2020, 3:40 PMAh yeah, should have clarified! I'm running on the most recent commit to
mastermasterj
Jim Crist-Harif
06/23/2020, 3:41 PMHmmm, things should run fine as configured. Do you have a reproducible example?
The docstrings might be incorrect in a few places (I might have missed a few), but the code should use the configured executor.
j
james.lamb
06/23/2020, 3:42 PMI can try to make one. Also maybe I got the wrong commit in my links. If you tell me "it's definitely expected that you can mix a
KubernetesJobEnvironmentj
Jim Crist-Harif
06/23/2020, 3:44 PMIt is expected, but there might still be bugs. If you want to try things out it'd be useful to get some external validation.
j
james.lamb
06/23/2020, 3:45 PMOk sure, thanks! It looks like those links I posted above are NOT from the latest commit. Bad click on my part.
I can see on
masterEnvironment.run_flow()KubernetesJobEnvironmentexecutorok let me try this again and check the places where I'm pegging to the latest commit. Maybe I made a mistake rebuilding the agent image. Sorry for the confusion!
it's super useful just to get confirmation that what I'm trying should work, so thank you for the quick response!
j
Jim Crist-Harif
06/23/2020, 3:47 PMNo worries, thanks for trying things out! We're definitely trying to simplify the deployment configuration, so please speak up as you find rough edges.
j
james.lamb
06/24/2020, 6:16 AMI think I found the root cause of the behavior I was seeing!
https://github.com/PrefectHQ/prefect/pull/2849