Elliott Wilson

01/17/2023, 12:40 PM
Hey team I am using a github storage block which is reading from our organisations private repo. I am using a PAT key to access. The prefect flows are nested in
and it seems that it can not pull the flow from the deployment when using a path. I have also created a new repo which has the deployment and the flow in the root directory and this works with the same PAT token. I am not sure if its an issue with the nested path or permissions on the file. Does any one have any ideas? I have included the deployment and stack trace, in the thread.
Copy code
### A complete description of a Prefect Deployment for flow 'mixpanel-to-s3'
name: mixpanel_to_s3
description: This flow extracts mixpanel data from the api and sends it to an s3 bucket.
version: 9f231da05569c0499e2e5b59d36049fa
# The work queue that will handle this deployment's runs
work_queue_name: default
work_pool_name: null
work_pool_queue_name: null
tags: []
parameters: {}
  cron: 0 2 * * *
  timezone: Europe/London
  day_or: true
infra_overrides: {}

flow_name: mixpanel-to-s3
manifest_path: null
  type: docker-container
    EXTRA_PIP_PACKAGES: mixpanel_utils==2.2.5 prefect_aws==0.2.1 s3fs
  labels: {}
  name: null
  command: null
  image: prefecthq/prefect:2.7.8-python3.9
  image_pull_policy: null
  image_registry: null
  networks: []
  network_mode: null
  auto_remove: true
  volumes: []
  stream_output: true
  memswap_limit: null
  mem_limit: null
  privileged: false
  _block_document_name: mixpanel-s3-docker-container
  _is_anonymous: false
  block_type_slug: docker-container
  _block_type_slug: docker-container
  repository: <>
  reference: main
  access_token: '**********'
  _block_document_name: github-monorepo
  _is_anonymous: false
  block_type_slug: github
  _block_type_slug: github
path: /data/prefect
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
timestamp: '2023-01-17T12:24:47.396515+00:00'
Copy code
WARNING: You are using pip version 21.3.1; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
/usr/local/lib/python3.9/ RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
12:28:50.491 | INFO    | Flow run 'enlightened-jerboa' - Downloading flow code from storage at 'monorepo/data/prefect'
12:28:53.081 | ERROR   | Flow run 'enlightened-jerboa' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/", line 266, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/", line 174, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/usr/local/lib/python3.9/site-packages/prefect/", line 922, in get_directory
    copytree(src=content_source, dst=content_destination, dirs_exist_ok=True)
  File "/usr/local/lib/python3.9/", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmphajvvwbjprefect/monorepo/data/prefect'
09:28:54.436 | INFO    | prefect.infrastructure.docker-container - Docker container 'enlightened-jerboa' has status 'removing'
09:28:54.445 | INFO    | prefect.infrastructure.docker-container - Docker container 'enlightened-jerboa' has status 'removing'
@Sahil Rangwala / @Bianca Hoch

Bianca Hoch

01/23/2023, 3:34 PM
Hey Elliott, here's the repo that I was referencing when walking through the deployment yaml
My second thought is that since the repo you're using for storage is private, it may be a matter of getting the permissions squared away.
Here's an example for setting those access tokens in a block for a private github repository, just for your reference.

Elliott Wilson

01/26/2023, 6:01 PM
Hey @Bianca Hoch and @Sahil Rangwala I tried the suggested work around and I am getting the same error. I am going to raise and issue for this.