Ofir03/01/2023, 7:14 PM
or any other 3rd party dependency from my workflow, what is the best practice to do that? The agent running my deployment/workflow might not have these packages in place, right? Should I build a custom agent Dockerfile with all of the dependencies, or is there a better approach to it? what are the tradeoffs between the different solutions? (edited)
Ryan Peden03/01/2023, 7:31 PM
is convenient for testing, but I recommend building dependencies into your image when running in production because installing dependencies at runtime adds significant overhead to each flow run. Nearly everything mentioned about
also applies to other infrastructure blocks that uses Docker containers, such as
for Google Cloud. One difference is that it's fine to use Kubernetes or cloud infrastructure from a containerized agent. Feel free to ask for more detail on anything I covered here; I mentioned a lot of concepts, and it's okay if you aren't familiar with all of them. 🙂
Ofir03/01/2023, 7:34 PM
Prefect agents rely on Docker images for executing flow runs usingor
DockerContainerinfrastructure. If you do not specify an image, we will use a Prefect image tag that matches your local Prefect and Python versions. If you are building your own image, you may find it useful to use one of the Prefect images as a base.
Ryan Peden03/01/2023, 7:52 PM
is built into the
package, your agents shouldn't need any extra dependencies. 2. Then, create a custom image that's only used for running flows. Your agents will use the
block to create a separate K8s job for each flow, and you can tell specify the name of the image to use. a. You can install all your dependencies and (optionally) copy your flow code in the image. By default, Prefect looks in
, but you can specify your own path if needed. b. If you don't want to copy your code into the image, you'll need to set up Storage for your deployments. Then, when the K8s job for your flow starts up, Prefect will download your code into the flow container and run it.
Ofir03/01/2023, 7:55 PM
Ryan Peden03/01/2023, 7:55 PM