https://prefect.io logo
Powered by
Title
a

Adam

04/14/2023, 8:56 PM
I am having an issue with an auth token for git. I have the below in my prefect.yaml for my project. It seems to be having an issue authenticating and wondering what I may be doing wrong that it cant get the right authentication 
pull:
- prefect.projects.steps.git_clone_project:
    repository: <https://github.com/comp_rep/bi_etl.git>
    branch: DATA-1425-docker-project-deploy
    access_token: "{{ prefect.blocks.github-credentials.git-creds }}"
✅ 1
for more context... looks like its trying to use a password... 
Cloning into 'etl'...
remote: Support for password authentication was removed on August 13, 2021.
remote: Please see <https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls> for information on currently recommended modes of authentication.
fatal: Authentication failed for '<https://github.com/comp_rep/etl.git/>'
a

alex

04/17/2023, 7:45 PM
Hey @Adam! That step expects a string for the access token so I would suggest using a 
Secret
block to store and reference your GitHub access token. We haven’t added support for other credentials blocks yet.
a

Adam

04/18/2023, 1:09 PM
ahhh ok, I was following the guide and saw aws creds and figured I could add the git ones, wasnt aware other blocks were not active yet. Thanks for getting back to me!
🙌 1
@alex The secret block worked! one more question, I have both yaml files, dockerfile and py file that runs the thing im automating. The only problem is it cant seem to find the py file. I get 
FileNotFoundError: [Errno 2] No such file or directory: '/app/etl/docker_prefect_test.py'
This is because the file path is supposed to be 
/app/etl/prefect_orchestration/docker_prefect_test.py
. I cant figure out where it is trying to retrieve this file so I can tell it do go one file down instead of the main repo area
a

alex

04/18/2023, 4:23 PM
Can you share your Dockerfile, your prefect.yaml file, and your deployment.yaml file?
a

Adam

04/18/2023, 4:24 PM
yeah, would you like them as attach or blocks in here?
a

alex

04/18/2023, 4:24 PM
Code blocks work!
a

Adam

04/18/2023, 4:25 PM
prefect.yaml 
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: prefect
prefect-version: 2.10.3

# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: prefect_orchestration/etl_orchestration
    tag: testing
    dockerfile: dockerfile_v2
    push: false

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.git_clone_project:
    repository: <https://github.com/comp_rep/etl.git>
    branch: DATA-1425-docker-project-deploy
    access_token: "{{ prefect.blocks.secret.git-access-token }}"
dockerfile 
# Use the official Python 3.10 slim base image
FROM python:3.10-slim

# Set the working directory to /app
WORKDIR /app

# Copy the requirements file into the container
COPY requirements.txt .

# Install any dependencies specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
RUN apt-get update
RUN apt-get install -y git

# Copy the rest of the application code into the container
COPY . .

# Set the default command to run when the container starts
CMD [ "python", "docker_prefect_test.py" ]
deployment.yaml 
# base metadata
name: null
description: null
tags: []
version: null
schedule: null

# flow-specific fields
entrypoint: prefect_orchestration/docker_prefect_test.py:run_dbt
path: null
parameters: {}

# infra-specific fields
work_pool:
  name: "docker-pool"
  work_queue_name: "docker-pool"
  job_variables:
    image: '{{ image_name }}'
a

alex

04/18/2023, 4:30 PM
Hmmm, everything looks good there. Can you share the full log output from your failed flow run?
a

Adam

04/18/2023, 4:31 PM
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/app/bi_etl/docker_prefect_test.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 217, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'docker_prefect_test.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
it seems like its reading from somewhere unintended to get to '/app/bi_etl/docker_prefect_test.py' instead of a specific folder
everytime i deploy it drops this in my flows.json 
"My dbt Flow": "docker_prefect_test.py:run_dbt"
wondering if that is causing it
a

alex

04/18/2023, 4:34 PM
Are you running 
prefect deploy
from the root of your project?
a

Adam

04/18/2023, 4:35 PM
no, I am running it from the folder where all the objects exist 
prefect_orchestration
otherwise it errors because it cant find the yamls
everything is in that folder
a

alex

04/18/2023, 4:42 PM
OK, that explains the difference between your configured entrypoint and the one that gets registered in 
flow.json
, but I wouldn’t expect that to cause this issue. I’m looking at our code for projects to figure out where your flow path might be getting messed up.
a

Adam

04/18/2023, 4:43 PM
awesome thank you! I didnt think there were that many places it could be looking and I thought I had them pointed at the folder/file
ill keep tinkering to see if I can figure it out as well
a

alex

04/18/2023, 4:48 PM
When you deploy, are you running just 
prefect deploy
with no options?
Actually, can you run 
prefect deployment inspect
with the name of this deployment and share the output?
a

Adam

04/18/2023, 4:53 PM
Missing argument 'NAME'.
and i have options when I run the deploy, should I run it without options?
a

alex

04/18/2023, 4:55 PM
Based on your deployment.yaml you’ll at least need to specify a name. You can see the names of your deployments with 
prefect deployments ls
and then run 
prefect deployment inspect
with the name that matches the deployment that we’re troubleshooting (e.g. 
prefect deployment inspect foo/bar
)
a

Adam

04/18/2023, 4:57 PM
I assume thats the name at the top in the metadata?
{
    'id': 'c5deeba1-55e8-4149-b005-d956b6d86309',
    'created': '2023-04-14T17:14:38.874963+00:00',
    'updated': '2023-04-18T16:58:40.172828+00:00',
    'name': 'first-project-test',
    'version': None,
    'description': None,
    'flow_id': '12594fe1-b35f-4e8c-adfb-89a34de8cf12',
    'schedule': None,
    'is_schedule_active': True,
    'infra_overrides': {'image': 'prefect_orchestration/etl_orchestration:testing'},
    'parameters': {},
    'tags': [],
    'work_queue_name': 'docker-pool',
    'parameter_openapi_schema': {'type': 'object', 'title': 'Parameters', 'properties': {}},
    'path': None,
    'pull_steps': [
        {
            'prefect.projects.steps.git_clone_project': {
                'branch': 'DATA-1425-docker-project-deploy',
                'repository': '<https://github.com/comp_rep/bi_etl.git>',
                'access_token': '{{ prefect.blocks.secret.git-access-token }}'
            }
        }
    ],
    'entrypoint': 'docker_prefect_test.py:run_dbt',
    'manifest_path': None,
    'storage_document_id': None,
    'infrastructure_document_id': None,
    'created_by': {'id': 'aacc29f0-0309-4f42-a9a7-14d6755a40b7', 'type': 'USER', 'display_value': 'adamdorenfeldthreeflowcom'},
    'updated_by': {'id': 'aacc29f0-0309-4f42-a9a7-14d6755a40b7', 'type': 'USER', 'display_value': 'adamdorenfeldthreeflowcom'},
    'work_pool_name': 'docker-pool'
}
there is the inspect
:gratitude-thank-you: 1
a

alex

04/18/2023, 5:03 PM
OK, the entrypoint here is different than the one configured in your deployment.yaml. I think these issues are stemming from the prefect files not being at the root of the repository. Could you try moving the prefect.yaml file, deployment.yaml file, and .prefect folder to the root of your repository and then running 
prefect deploy
from the root of your repository?
a

Adam

04/18/2023, 5:05 PM
i was running this to deploy, should I keep these options? 
prefect deploy docker_prefect_test.py:run_dbt \
    -n first-project-test \
    -p docker-pool
also if this does work, would I just have to clog the root with a new yaml for every deployment I want to use?
better question, How does one make more than 1 deployment through this method just many yamls in root?
looks like I have to move all files out and put them in root
a

alex

04/18/2023, 5:09 PM
Currently, we only support one deployment per project via the 
deployment.yaml
file. Additional deployments would need to be done 100% with the CLI. However, we are looking to introduce the ability to declare multiple deployments in a project with this PR which should make deployment management much simpler.
Once you move those file you can update your command to this 
prefect deploy -n first-project-test
a

Adam

04/18/2023, 5:09 PM
I dont mind doing more than one deployment yaml
just wondering if they all have to be in root and cant be stored in their own folders
a

alex

04/18/2023, 5:11 PM
Well, we only support one 
deployment.yaml
file for each project and it’s assumed to be at the root of the project.
a

Adam

04/18/2023, 5:11 PM
looks like I have to move everything to root, reqirement.txt and dockerfile as well
so 1 repo 1 deployment
if thats the case I may go back to using the Ib and storage blocks, that seemed to be working pretty well
a

alex

04/18/2023, 5:12 PM
In either the next release, or the one after, you’ll be able to do multiple deployments per repo.
👏 1
a

Adam

04/18/2023, 5:12 PM
I do like the structure of the projects
ok so the requirements, docker, and both yaml have to be in root it seems, testing that now
it ran!
a

alex

04/18/2023, 5:14 PM
I think if you update your 
build
step to point to a Dockerfile nested in the directory then you should be able to keep it and the requirements.txt file where they are.
a

Adam

04/18/2023, 5:14 PM
ok cool, I will try that out, at the very least it works when everything is in root!
and I REALLY appreciate you walking me through all this!
a

alex

04/18/2023, 5:15 PM
I really appreciate you hanging in there with me!
a

Adam

04/18/2023, 5:16 PM
ok so what we learned is all things in root, and be patient for multiple deploys. I dont need multiple right now so its all good, was planning for future work
👍 1
now I get to attempt to do the push section to ECS 🤣
last note... updating the dockerfile and build worked so only the yamls needed to be outside
🙌 1
#prefect-communityJoin Slack
Powered by