Hey guys, been working on migrating to ECS push work pools and I’m struggling to understand this error. Context: I have a custom docker image built and pushed to ECR as part of my CI/CD. Image contains my scripts as well as all my dependencies installed. WORKDIR is set to

opt/prefect/

. Image is meant to be used for all my deployments so no ENTRYPOINT or CMD is set. Deployment is done with a prefect.yaml file with just the build (that references the Dockerfile), push and deployments section. CI/CD runs fine and my flow gets deployed. This error appears when I try running the deployed flow. I’ve reached the try-anything-and-see-what-works phase. Any clue what I should do? 🙏

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 426, in retrieve_flow_then_begin_flow_run
    else await load_flow_from_flow_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 290, in load_flow_from_flow_run
    basepath = deployment.path or Path(deployment.manifest_path).parent
  File "/usr/local/lib/python3.10/pathlib.py", line 960, in __new__
    self = cls._from_parts(args)
  File "/usr/local/lib/python3.10/pathlib.py", line 594, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/local/lib/python3.10/pathlib.py", line 578, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

hey @Sarhan - can you share the entire trace?

Hey @Nate, thanks for getting back. That’s actually everything that gets logged in the prefect cloud UI. Any other place I should be looking to see a longer trace? Or any configuration I can change to enhance what gets logged?

hrm yeah that’s not the best trace, can you show your deployment definition?

Sure @Nate, here you go:

deployments:
  - name: test-flow-deployment
    entrypoint: scripts/flows/test_flow.py:test_flow
    tags: [test]
    work_pool:
      name: default-ecs-work-pool
      work_queue_name: default-push-queue
      job_variables:
        image: "{{ build-image.image }}"

hmmmm, looks pretty normal are you able to share

prefect version

output and your flow definition? feel free to DM if you dont want to share here

Sure, will DM

Hi, was anyone able to come to a resolution to this issue? Thanks

hi @Anthony W - if i remember correct, there was a configuration issue that wasn't on the prefect side are you migrating from agents to workers by chance?

No, I am attempting to setup a push ECS work pool. I am not selecting 'yes' in the deployment step for creating a custom image. My work pool and ECS task both reference a specified image. Here is the deployment yaml that was created after creating the deployment via the prefect cli:

- name: test_flow
  version:
  tags: []
  description:
  entrypoint: /flows/test_flows/test_flow.py:test_flow
  parameters: {}
  work_pool:
    name: ecs-dev
    work_queue_name: default
    job_variables: {}
  schedules: []

Also, since this looks to be an issue with Prefect not finding the flow code, here is my dockerfile:

FROM python:3.12.3-slim
WORKDIR /app

# Package prep
...

# Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN export PYTHONPATH=/app

okay. where should your flow code come from? i dont see in the above whether you expect it to come from • a

pull

step at runtime • a

COPY

into your image at deployment time

Sorry, here's the top portion of the deployment yaml:

pull:
- prefect.deployments.steps.git_clone:
    repository: <https://gitlab.repolinkhere.git>
    branch: dev
    access_token: '{{ prefect.blocks.secret.-repo-token
      }}'

is that verbatim what you have? that repo token secret block name looks weird

'{{ prefect.blocks.secret.-repo-token
      }}'

No, it is intentionally obfuscated

gotcha, do you have logs from failed flow runs you've seen? im not sure yet, logs seem like the best thing to check next

There is only one entry in the Prefect Cloud logs:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 218, in asyncnullcontext
    yield
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments.py", line 155, in load_flow_from_flow_run
    basepath = deployment.path or Path(deployment.manifest_path).parent
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 871, in __new__
    self = cls._from_parts(args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 509, in _from_parts
    drv, root, parts = self._parse_args(args)
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 493, in _parse_args
    a = os.fspath(a)
        ^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType

oohhh, that version is probably the issue. workers were introduced in 2.10 iirc. can you try a newer 2.x version like 2.20.x?

@Sarhan, @Anthony W, @Nate did you ever resolve this issue? I'm also getting the

Flow could not be retrieved from deployment.

Its random when it happens but we are also using Kubernetes (EKS w/ Fargate Nodes), and the code exists on the custom Docker Container image we are using so no code needs to be pulled in. > Flow could not be retrieved from deployment. > Traceback (most recent call last): > File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 246, in retrieve_flow_then_begin_flow_run > flow = await load_flow_from_flow_run(flow_run, client=client) > File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 105, in with_injected_client > return await fn(*args, **kwargs) > File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 50, in load_flow_from_flow_run > basepath = deployment.path or Path(deployment.manifest_path).parent > File "/usr/local/lib/python3.9/pathlib.py", line 1082, in new > self = cls._from_parts(args, init=False) > File "/usr/local/lib/python3.9/pathlib.py", line 707, in _from_parts > drv, root, parts = self._parse_args(args) > File "/usr/local/lib/python3.9/pathlib.py", line 691, in _parse_args > a = os.fspath(a) > TypeError: expected str, bytes or os.PathLike object, not NoneType > 041237 PM > prefect.flow_runs Deployment_Prefect.yaml

prefect-version: 3.0.5
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /code
deployments:
- name: null
  version: null
  tags: []
  description: null
  schedule: {}
  flow_name: null
  entrypoint: null
  parameters: {}
  work_pool:
    name: work-pool-dev
    work_queue_name: eks-queue
    job_variables:
      image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/mycontainer:latest
      env: 
        APP_ENV: dev

poetry run prefect deploy /code/myflow.py:myflow -n dev -q eks-queue -p work-pool-dev --prefect-file /Deployment_Prefect.yaml