Adam
04/14/2023, 8:56 PMpull:
- prefect.projects.steps.git_clone_project:
repository: <https://github.com/comp_rep/bi_etl.git>
branch: DATA-1425-docker-project-deploy
access_token: "{{ prefect.blocks.github-credentials.git-creds }}"
Cloning into 'etl'...
remote: Support for password authentication was removed on August 13, 2021.
remote: Please see <https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls> for information on currently recommended modes of authentication.
fatal: Authentication failed for '<https://github.com/comp_rep/etl.git/>'
alex
04/17/2023, 7:45 PMSecret
block to store and reference your GitHub access token. We haven’t added support for other credentials blocks yet.Adam
04/18/2023, 1:09 PMFileNotFoundError: [Errno 2] No such file or directory: '/app/etl/docker_prefect_test.py'
This is because the file path is supposed to be /app/etl/prefect_orchestration/docker_prefect_test.py
. I cant figure out where it is trying to retrieve this file so I can tell it do go one file down instead of the main repo areaalex
04/18/2023, 4:23 PMAdam
04/18/2023, 4:24 PMalex
04/18/2023, 4:24 PMAdam
04/18/2023, 4:25 PM# File for configuring project / deployment build, push and pull steps
# Generic metadata about this project
name: prefect
prefect-version: 2.10.3
# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.1.0
image_name: prefect_orchestration/etl_orchestration
tag: testing
dockerfile: dockerfile_v2
push: false
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.git_clone_project:
repository: <https://github.com/comp_rep/etl.git>
branch: DATA-1425-docker-project-deploy
access_token: "{{ prefect.blocks.secret.git-access-token }}"
# Use the official Python 3.10 slim base image
FROM python:3.10-slim
# Set the working directory to /app
WORKDIR /app
# Copy the requirements file into the container
COPY requirements.txt .
# Install any dependencies specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
RUN apt-get update
RUN apt-get install -y git
# Copy the rest of the application code into the container
COPY . .
# Set the default command to run when the container starts
CMD [ "python", "docker_prefect_test.py" ]
# base metadata
name: null
description: null
tags: []
version: null
schedule: null
# flow-specific fields
entrypoint: prefect_orchestration/docker_prefect_test.py:run_dbt
path: null
parameters: {}
# infra-specific fields
work_pool:
name: "docker-pool"
work_queue_name: "docker-pool"
job_variables:
image: '{{ image_name }}'
alex
04/18/2023, 4:30 PMAdam
04/18/2023, 4:31 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap_external>", line 1016, in get_code
File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/app/bi_etl/docker_prefect_test.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 217, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 201, in import_object
module = load_script_as_module(script_path)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'docker_prefect_test.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
"My dbt Flow": "docker_prefect_test.py:run_dbt"
wondering if that is causing italex
04/18/2023, 4:34 PMprefect deploy
from the root of your project?Adam
04/18/2023, 4:35 PMprefect_orchestration
otherwise it errors because it cant find the yamlsalex
04/18/2023, 4:42 PMflow.json
, but I wouldn’t expect that to cause this issue. I’m looking at our code for projects to figure out where your flow path might be getting messed up.Adam
04/18/2023, 4:43 PMalex
04/18/2023, 4:48 PMprefect deploy
with no options?prefect deployment inspect
with the name of this deployment and share the output?Adam
04/18/2023, 4:53 PMMissing argument 'NAME'.
alex
04/18/2023, 4:55 PMprefect deployments ls
and then run prefect deployment inspect
with the name that matches the deployment that we’re troubleshooting (e.g. prefect deployment inspect foo/bar
)Adam
04/18/2023, 4:57 PM{
'id': 'c5deeba1-55e8-4149-b005-d956b6d86309',
'created': '2023-04-14T17:14:38.874963+00:00',
'updated': '2023-04-18T16:58:40.172828+00:00',
'name': 'first-project-test',
'version': None,
'description': None,
'flow_id': '12594fe1-b35f-4e8c-adfb-89a34de8cf12',
'schedule': None,
'is_schedule_active': True,
'infra_overrides': {'image': 'prefect_orchestration/etl_orchestration:testing'},
'parameters': {},
'tags': [],
'work_queue_name': 'docker-pool',
'parameter_openapi_schema': {'type': 'object', 'title': 'Parameters', 'properties': {}},
'path': None,
'pull_steps': [
{
'prefect.projects.steps.git_clone_project': {
'branch': 'DATA-1425-docker-project-deploy',
'repository': '<https://github.com/comp_rep/bi_etl.git>',
'access_token': '{{ prefect.blocks.secret.git-access-token }}'
}
}
],
'entrypoint': 'docker_prefect_test.py:run_dbt',
'manifest_path': None,
'storage_document_id': None,
'infrastructure_document_id': None,
'created_by': {'id': 'aacc29f0-0309-4f42-a9a7-14d6755a40b7', 'type': 'USER', 'display_value': 'adamdorenfeldthreeflowcom'},
'updated_by': {'id': 'aacc29f0-0309-4f42-a9a7-14d6755a40b7', 'type': 'USER', 'display_value': 'adamdorenfeldthreeflowcom'},
'work_pool_name': 'docker-pool'
}
alex
04/18/2023, 5:03 PMprefect deploy
from the root of your repository?Adam
04/18/2023, 5:05 PMprefect deploy docker_prefect_test.py:run_dbt \
-n first-project-test \
-p docker-pool
alex
04/18/2023, 5:09 PMdeployment.yaml
file. Additional deployments would need to be done 100% with the CLI. However, we are looking to introduce the ability to declare multiple deployments in a project with this PR which should make deployment management much simpler.prefect deploy -n first-project-test
Adam
04/18/2023, 5:09 PMalex
04/18/2023, 5:11 PMdeployment.yaml
file for each project and it’s assumed to be at the root of the project.Adam
04/18/2023, 5:11 PMalex
04/18/2023, 5:12 PMAdam
04/18/2023, 5:12 PMalex
04/18/2023, 5:14 PMbuild
step to point to a Dockerfile nested in the directory then you should be able to keep it and the requirements.txt file where they are.Adam
04/18/2023, 5:14 PMalex
04/18/2023, 5:15 PMAdam
04/18/2023, 5:16 PM