https://prefect.io logo
g

Grant

09/01/2023, 3:20 PM
When running prefect deploy from a terminal, I'm no longer being asked github repo information, and trying to manually update the prefect.yaml file with my repo to clone into is not working. Was there an update recently that changed the github connection aspect of prefect deployments?
a

alex

09/01/2023, 3:23 PM
Hey @Grant! Can you share the traceback of the error you’re seeing if you have one?
g

Grant

09/01/2023, 3:27 PM
@alex The error is just a "FileNotFoundError" because the prefect deployment is telling the workers to load the flow from a local working directory rather than my github repo, see below. Was the feature where prefect prompts me to enter github information in the deployment commands removed?
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "c:\Users\Grant.Arbuckle\Prefect\Data-Pipeline-Flows\.venv\Lib\site-packages\prefect\deployments\steps\core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Grant.Arbuckle\Prefect\Data-Pipeline-Flows\.venv\Lib\site-packages\prefect\deployments\steps\core.py", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Grant.Arbuckle\Prefect\Data-Pipeline-Flows\.venv\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 292, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Grant.Arbuckle\Prefect\Data-Pipeline-Flows\.venv\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\Grant.Arbuckle\Prefect\Data-Pipeline-Flows\.venv\Lib\site-packages\prefect\deployments\steps\pull.py", line 28, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\grant.arbuckle\\OneDrive - Steel Dynamics Inc\\Desktop\\Python Files\\Data-Pipeline-Flows'
a

alex

09/01/2023, 4:08 PM
There hasn’t been a change in the prompt behavior for git repos in a couple of months, but we only show that prompt if we can detect that
prefect deploy
was run in a local git repo and there isn’t already a pull step configured in your
prefect.yaml
. Can you share your
prefect.yaml
file?
g

Grant

09/01/2023, 4:21 PM
I'm confused then why I was previously getting the github prompt and now I'm not. I am deploying from the same location, in a cloned git repo directory. Here is the prefect.yaml file when I run prefect deploy:
Copy code
# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: Data-Pipeline-Flows
prefect-version: 2.12.0

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: C:\Users\grant.arbuckle\OneDrive - Steel Dynamics Inc\Desktop\Python
      Files\Data-Pipeline-Flows

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: update_qa_salesforce_tables
  version: null
  tags: []
  description: null
  entrypoint: salesforce_staging_qa\salesforce_staging_qa_flow.py:update_qa_salesforce_tables
  parameters: {}
  work_pool:
    name: cw_datapro_pool
    work_queue_name: null
    job_variables: {}
  schedule:
    cron: 16 * * * *
    timezone: EST
    day_or: true
@alex But even when I change the pull section to my repo and save that change, the deployment doesn't pick up on it and still tries to pull the flow code from a local working directory:
Copy code
# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: Data-Pipeline-Flows
prefect-version: 2.12.0

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/steeldynamics/Data-Pipeline-Flows.git>
    branch: All-Flows

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: update_qa_salesforce_tables
  version: null
  tags: []
  description: null
  entrypoint: salesforce_staging_qa\salesforce_staging_qa_flow.py:update_qa_salesforce_tables
  parameters: {}
  work_pool:
    name: cw_datapro_pool
    work_queue_name: null
    job_variables: {}
  schedule:
    cron: 16 * * * *
    timezone: EST
    day_or: true
a

alex

09/01/2023, 4:43 PM
Have you run
prefect deploy
again after updating your
prefect.yaml
? The steps in the
pull
section must be saved via the Prefect API for the change to take effect in your deployments.
g

Grant

09/01/2023, 4:49 PM
Yes I have, and I keep getting this message whenever I walk through the prefect deploy prompts even after updating the prefect.yaml (see image). I also included an image of my directory hierarchy, why would this suddenly be an issue when it hasn't been an issue for any of the the other flows I've deployed?
a

alex

09/01/2023, 4:50 PM
Hmm, that surprises me. Are you seeing a prompt to select from your existing deploy configs when you run
prefect deploy
?
g

Grant

09/01/2023, 4:51 PM
image.png
I get a prompt to "Select a flow to deploy", but no prompt to select from existing deploy configs:
It seems like something changed and prefect is not picking up on the fact that I'm working in a git repo now, looks like trying to redeploy my other flows has the same result - trying to run code from a local working directory instead of a repo
a

alex

09/01/2023, 4:55 PM
Gotcha. In order to see a prompt to select from existing deployments you’ll need to run
prefect deploy
in the same directory as a
prefect.yaml
file. I notice that you have multiple
prefect.yaml
files. We generally recommend having one
prefect.yaml
file per project/repository with all the deploy configs in that one file. That would allow you to deploy any flow in the entire project from the project/repo root.
👍 1
g

Grant

09/01/2023, 4:57 PM
Gotcha, I'll try adding all the deployments to the same prefect.yaml file
@alex after reworking my directory to have all deployments contained within a single prefect.yaml file, that resolved the issue! I didn't know having just one .yaml file was best practice. Thank you for your help and patience with me! 🙏