having a hard time understanding deployments. i setup everything up, deploy to my internal server and the flow just keeps crashing. Completed submission of flow run '37064233-65e1-4a3c-b87e-656efe6d778d' 083421 AM prefect.flow_runs.worker ERROR Process 11612 exited with status code: 2 i think its because its not using a venv , isntalling packages, and activating it before running. can i get some guidance here? this is what my prefect.yaml looks like

name: minimal-windows-project

deployments: - name: minimal-flow-deployment entrypoint: "test_flow.py:hello_flow" work_pool: name: windows-pool path: "C:\\Tools\\Prefect\\storage\\Prefect_Server" build: - prefect.deployments.steps.run_python_script: script: "setup.py" - prefect.deployments.steps.pip_install_requirements: requirements: "requirements.txt" stream_output: true

hi @Kevin Mach

build:

- prefect.deployments.steps.run_python_script:

script: "setup.py"

this part seems like an issue, since I'm not sure that was ever a real deployment step (I could be wrong) could you share the output of

prefect version

?

thanks for the response @Nate Version: 3.0.0rc11 API version: 0.8.4 Python version: 3.11.4 Git commit: 6604ec94 Built: Wed, Jul 17, 2024 4:17 PM OS/Arch: win32/AMD64 Profile: default Server type: server Pydantic version: 2.8.2 Integrations: prefect-docker: 0.5.2

thanks! so 2 things i notice • you’ll need an rc version of prefect-docker, which you can get with pip install -U prefect-docker —pre • run_python_script is not an out of the box deployment step we offer, perhaps you’re looking for run_shell_script? what kind of worker would you like to use? ie docker, process etc

Yea docker . I have a lot of scripts that I wrote that have a lot of custom python modules . So I’d like to setup all that in a docker container . But one thing is due to limitations at work I have to use docker windows , can’t use docker with the WSL Linux subsystem . So when it tried to use prefect-docker it failed because it said it didn’t have an image for windows/amd64 Does my docker containers have to run on top of Unix?

sorry! im not so familiar with running prefect using windows images in case this is helpful (example run) ps not saying you should use GHA to run your worker 🙂 just showing how you can start one somewhere you run your long lived processes

@Nate so quick question, and thanks again for all your support thus far! appreciate it. the docker container should clone down the code base, and setup the python environment with requirements from pip requirements.txt? and then the prefect.yaml will call it? all i need to do is pass in the entrypoint?i have the work-pool and workers setup to accept docker already so is something like this good for the prefect.yaml? name: docket-hello-flow deployments: - name: docker-deployment entrypoint: test_flow.py:hello_flow work_pool: name: docker-pool work_queue_name: default build: - prefect_docker.deployments.steps.build_docker_image: image_name: hello-flow tag: latest dockerfile: Dockerfile pull: - prefect_docker.deployments.steps.pull_docker_image: image_name: hello-flow tag: latest infrastructure: type: docker-container env: PREFECT_API_URL: "http://10.20.30.83:4200/api" image: hello-flow:latest

sure thing - are you keeping your code in github/bitbucket and then wanting to clone it before runtime or are you planning to bake your source into the images? here are the docs on this

i think ideally id like to keep source code in gitlab, its a private one within the enterprise, but i had some issues getting the token and oath to work so ive been trying to bake the code into the image . and if i have a container , is there still a need to setup a venv and install dependencies to there? or since its already a container i can just do a regualr pip install -r requirements.txt with out using a venv?

can you share logs from the worker container? or is the above all you get back?

That’s all I get back . Been trying to find more logs but that’s all I see

> and if i have a container , is there still a need to setup a venv and install dependencies to there? it depends on whether the

image

you choose for your work pool (or for that deployment specifically with a job_variables override) has them installed already if you use an

image

that has your dependencies installed, then you need only clone your script from gitlab with

git_clone

Here are some common reasons for seeing exited with non-zero status code 1 in a setting where you submit a container via a

prefect-docker

worker: 1. Intermittent Docker Errors: Some deployments may crash due to intermittent Docker errors, often not consistently reproducible. Link 2. Module Errors: The Docker client failing to get initialized correctly, such as missing attributes in the Docker module. Link 3. Resource Constraints: High memory usage causing the container to be terminated by the operating system. Link 4. Timeouts: The container times out and gets terminated, leaving flow runs in an inconsistent state. Link You can explore more specific details and solutions by checking the links provided.

will check back tomorrow!

in my docker container everything is put in a apps folder. then i have the test_flow.py file right at the root. so my entry point is test_flow.py:hellow_flow if i run the docker container interactively i can run the python script manually and it works, it kicks off the flow on the remote server i have setup

i saw apache airflow and was like ehhh... lol

makes sense 🙂 there’s a few patterns i’ve recorded here over time in case they’re helpful

Thank you ! I worked on it more today and got deployments if it’s just a simple hello world file with tasks and flow defined . Right why my scripts get a bit more complex like using a dependency like pyodbc it fails when I try to deploy . But I can kick off locally . So I think I’m not configuring the prefect.yaml correctly