Registering my flows takes a (relatively) really long time, is there anything I can do to speed it u...

Zach

12/01/2020, 9:38 PM

Registering my flows takes a (relatively) really long time, is there anything I can do to speed it up

Jim Crist-Harif

12/01/2020, 9:40 PM

Hi Zach, without knowing more about what you're doing it's hard to answer this. What kind of flow storage/environment/run_config are you using?

Zach

12/01/2020, 9:46 PM

I am using the

KubernetesJobEnvironment

. For storage I am using

prefect.environments.storage.Docker

, I am not sure what you mean by run_config, I am using the

PrefectFlowManager

if that is what you mean, but I don't think it is.

Jim Crist-Harif

12/01/2020, 9:47 PM

We're moving to a new configuration system for flow-runs, configured by

flow.run_config

(see https://docs.prefect.io/orchestration/flow_config/overview.html) - these will replace environments (after a long deprecation cycle).

Jim Crist-Harif

12/01/2020, 9:50 PM

The main slowdown for you would then be

Docker

storage building. This is less a question about how to configure prefect, and more a question about how to speedup docker storage builds. A few tips: • If you have lots of custom dependencies, you might build a custom base image used by all your flows, and set this as

base_image

. Then your flow build would only need to add the flow source to the image before registering. • Alternatively you might use a different storage type (

S3

Github

, ...) for hosting your flow source, and configure a static image on your flow separately. To do that using the legacy environment-based system you'd set

flow.environment.metadata["image"] = "your-image-name"

). If you're using a

KubernetesRun

run-config, this would just be the

image

parameter to that object.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.