Kevin Mach
07/22/2024, 3:44 PMname: minimal-windows-project
deployments:
- name: minimal-flow-deployment
entrypoint: "test_flow.py:hello_flow"
work_pool:
name: windows-pool
path: "C:\\Tools\\Prefect\\storage\\Prefect_Server"
build:
- prefect.deployments.steps.run_python_script:
script: "setup.py"
- prefect.deployments.steps.pip_install_requirements:
requirements: "requirements.txt"
stream_output: trueNate
07/23/2024, 1:23 AMbuild:
- prefect.deployments.steps.run_python_script:
script: "setup.py"this part seems like an issue, since I'm not sure that was ever a real deployment step (I could be wrong) could you share the output of
prefect version
?Kevin Mach
07/23/2024, 1:37 AMNate
07/23/2024, 1:48 AMKevin Mach
07/23/2024, 2:37 AMKevin Mach
07/23/2024, 3:45 AMKevin Mach
07/23/2024, 3:53 AMKevin Mach
07/23/2024, 4:15 AMNate
07/23/2024, 4:19 AMNate
07/23/2024, 4:20 AMgit_clone
step and the pip_install_requirements
step if the former
or the set_working_directory
step if the latter
pull_docker_image
is not a pull
step that we includeKevin Mach
07/23/2024, 4:24 AMKevin Mach
07/23/2024, 4:26 AMNate
07/23/2024, 4:26 AMKevin Mach
07/23/2024, 4:27 AMNate
07/23/2024, 4:29 AMimage
you choose for your work pool (or for that deployment specifically with a job_variables override) has them installed already
if you use an image
that has your dependencies installed, then you need only clone your script from gitlab with git_clone
Nate
07/23/2024, 4:32 AMexited with non-zero status code 1in a setting where we submit a container via a
prefect-docker
worker? be concise and search issuesMarvin
07/23/2024, 4:32 AMprefect-docker
worker:
1. Intermittent Docker Errors: Some deployments may crash due to intermittent Docker errors, often not consistently reproducible. Link
2. Module Errors: The Docker client failing to get initialized correctly, such as missing attributes in the Docker module. Link
3. Resource Constraints: High memory usage causing the container to be terminated by the operating system. Link
4. Timeouts: The container times out and gets terminated, leaving flow runs in an inconsistent state. Link
You can explore more specific details and solutions by checking the links provided.Nate
07/23/2024, 4:32 AMKevin Mach
07/23/2024, 4:33 AMKevin Mach
07/23/2024, 4:35 AMKevin Mach
07/23/2024, 4:35 AMNate
07/24/2024, 3:30 AMi saw apache airflow and was like ehhh... lolmakes sense 🙂 there’s a few patterns i’ve recorded here over time in case they’re helpful
Kevin Mach
07/24/2024, 5:37 AMKevin Mach
07/24/2024, 5:38 AMKevin Mach
07/24/2024, 4:06 PM- prefect.deployments.steps.pip_install_requirements:
directory: '{{ clone-repo.directory }}'
requirements_file: requirements.txt
then i looked on line, i think someone said i need to add this to the deployment section?
steps:
- run:
command: pip install -r requirements.txt
label: Install dependencies
but either way it fails. any idea? my next route is to try to setup docker containers.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by