Hi team, I'm new to using prefect so bare with me...
# ask-community
e
Hi team, I'm new to using prefect so bare with me. I'm getting the attached error when trying to run the remote deploy (managed via github actions) from prefect cloud however the same code works when I use a local deploy. openai looks to be installed to me when looking at the generated logs. Furthermore I'm pretty sure this worked when I tried it yesterday.
n
hi @Emmanuel Quarless - that second picture where you show deps installed, where is that? relatedly, where / how are you specifying what dependencies should exist in your remote runtime?
e
Hi @Nate - that's the output in githb actions after running the "Prefect Deploy" step defined in the third picture (github actions workflow)
I don't think im specifying the dependencies needed for the remote runtime anywhere 🤔 where would that usually live?
n
right ok that’s what i suspected! so you essentially need to include some information in your deployment on runtime deps, and there’s a couple ways to do that depending on your runtime. what kind of work pool are you using?
installing the deps at deploy time is not strictly necessary but is recommended so we can build a parameter schema so you get nice forms in the UI, but generally speaking if you have a containerized runtime (docker k8s ECS etc) then the typical thing is to build an image (during prefect deploy or separately, it’s up to you) and set that
image
tag as a job variable on your work pool @Marvin get related docs pls
m
Here's some relevant information on deploying Prefect flows with containerized runtimes and dependency management: 1. Dependencies at Deploy Time: It's recommended to install dependencies at deploy time to build a parameter schema, which enhances the UI experience with forms. However, if you're using a containerized runtime (like Docker, Kubernetes, or ECS), it's typical to build an image with all necessary dependencies and set that image as a job variable on your work pool. For more details, you can refer to Prefect YAML. 2. Building Docker Images: When using Docker, you can build the image using the
docker build
command and push it to a container registry. Prefect provides guidelines on how to manage this process, including injecting environment variables and running the container. More information is available in the Docker Deployment Guide. 3. Managing Work Pools: Work pools allow you to manage infrastructure for flow runs. You can version control your job templates and update them through CI/CD pipelines. Prefect supports various work pool types, and you can find more about them in the Work Pools Documentation. 4. Prefect Blocks: Blocks in Prefect store typed configurations and are useful for managing parameters and infrastructure settings across deployments. For more on blocks and their uses, see Prefect Blocks. If you have further questions or need more specific details, feel free to ask.
I couldn't find any specific GitHub issues related to building Docker images and setting job variables for work pools in Prefect. However, here are some relevant documentation links that might help you with this process: 1. Docker and Containerized Runtimes: Learn how to build and run Docker images for Prefect flows, including pushing images to a registry and configuring environment variables. Check the Docker Deployment Guide. 2. Work Pools: Understand how to manage work pools, including setting job variables and using different infrastructure types like Docker, Kubernetes, and ECS. More information is available in the Work Pools Documentation. If you have further questions or need more specific details, feel free to ask.
To manage dependencies and build images for Prefect deployments in a containerized runtime, here are some relevant resources and documentation: 1. Work Pools and Job Variables: Prefect's work pools use a base job template that includes a
job_configuration
section. This section can have variables that you customize per deployment. You can set the
image
tag as a job variable in this configuration. More details can be found in the Work Pools Documentation. 2. Building Docker Images: Prefect provides guidance on building Docker images using the
docker build
command. You can specify a Dockerfile and include dependencies in a
requirements.txt
file. The image can then be pushed to a container registry. Refer to the Docker Deployment Guide for more details. 3. Custom Docker Images: If you need a custom Dockerfile, Prefect allows you to specify it using the
DockerImage
class. This lets you customize the image further, such as installing private packages. Learn more in the Docker Infrastructure Examples. These resources should help you understand how to build and manage Docker images for Prefect deployments and set them as job variables in your work pools.
n
oops i triggered marvin again bc i edited my message 🙂
e
thanks for getting back to be btw! so in my that last pip install in my third image wouldn't do the job? and the workpool is
Prefect:managed
I kinda wanted to avoid any Dockerisation https://docs.prefect.io/v3/deploy/infrastructure-concepts/deploy-ci-cd#build-deployments-via-ci-cd
n
oh yeah if you’re using the managed pool things are easier check out this example • define your flow • define your deployment in a `prefect.yaml` • use the cli to deploy (you can do this in CI as you are now)
Copy code
» prefect --no-prompt deploy -n uses-managed-work-pool
• trigger and observe!
Copy code
» prefect deployment run 'load-titanic-dataset/uses-managed-work-pool' --watch
Creating flow run for deployment 'load-titanic-dataset/uses-managed-work-pool'...
Created flow run 'resilient-fox'.
└── UUID: 54686b95-0be3-4f0e-80fc-9b3a184d6e31
└── Parameters: {}
└── Job Variables: {}
└── Scheduled start time: 2025-02-07 19:28:12 CST (now)
└── URL: <https://app.prefect.cloud/account/xxx/workspace/cafa2ffa-f6cc-4ed6-ab76-eaa4ba1ad40e/runs/flow-run/xxx>
Watching flow run 'resilient-fox'...
19:28:13.202 | INFO    | prefect - Flow run is in state 'Scheduled'
19:28:18.304 | INFO    | prefect - Flow run is in state 'Scheduled'
19:28:23.419 | INFO    | prefect - Flow run is in state 'Scheduled'
19:28:28.523 | INFO    | prefect - Flow run is in state 'Pending'
19:28:33.633 | INFO    | prefect - Flow run is in state 'Pending'
19:28:38.734 | INFO    | prefect - Flow run is in state 'Pending'
19:28:43.877 | INFO    | prefect - Flow run is in state 'Pending'
19:28:49.000 | INFO    | prefect - Flow run is in state 'Pending'
19:28:54.098 | INFO    | prefect - Flow run is in state 'Pending'
19:28:59.185 | INFO    | prefect - Flow run is in state 'Completed'
Flow run finished successfully in 'Completed'.
🙌 1
but if you have a bunch of special deps, i would bite the bullet and
build
an
image
like the other examples in that repo so you can set that
image
as one of the
job_variables
for that deployment
e
ahh I follow you now thanks! Ye defining dependencies in the prefect.yaml file seems like it'll be a bit cumbersome so I'll probably take your advice and just build an image haha. (I'd also like to confirm it works once i put openai under the work pool's pip dependencies 🔥)
catjam 1