https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Sometimes in a scheduled deployment a...
# ask-marvin
m
@Marvin Sometimes in a scheduled deployment a worker will never submit a flow run to ecs. The resulting run will promptly error with the log. "Flow can not be retrieved from deployment". Why does this happen?
m
✅ thought for 385.3 seconds
Short answer: this happens when Prefect can’t load your flow code from the deployment at run time, so the worker never submits anything to ECS and the run errors quickly with “Flow can not be retrieved from deployment.” Why this occurs (most common causes) - Invalid or missing entrypoint in the deployment - The deployment record must contain a valid 
entrypoint
like 
path/to/file.py:flow_name
or 
package.module.flow_name
. If it’s wrong or missing, Prefect can’t load the flow and will not submit to ECS. - Source/pull step can’t fetch your code - Git clone fails due to missing/invalid credentials, the repo is private, the branch doesn’t exist anymore, or the referenced commit SHA is gone. - Using a local filesystem path as source but running in ECS (the code isn’t present in the container). - The entrypoint no longer matches the code - The file moved or the flow function was renamed/removed after you created the deployment (especially if you deploy from a moving branch). - Import errors at load time - The module imports fail in the runtime environment due to missing dependencies or syntax/import errors. Why it’s “sometimes” - If your deployment points to a moving branch, a scheduled run may pick up a new commit where the entrypoint/file/flow has changed or credentials/token expired. Pinning to a commit SHA stabilizes this. How to confirm and debug 1) Inspect the deployment to verify entrypoint and pull steps
Copy code
prefect deployment inspect "your-flow/your-deployment"
- Confirm 
entrypoint
is correct and present. - If you use Git, ensure your 
pull
steps reference the right repo/branch/commit and credentials block. 2) Prefer immutable versions - Pin deployments to a commit SHA instead of a branch. See Versioning deployments. 3) Check credentials and pull steps - If using Git, make sure a credentials block is configured and referenced correctly. Docs: Pull steps API. 4) Validate the entrypoint locally - From the same environment you use to build/deploy, try importing the flow by its entrypoint (e.g., 
from src.flows.my_flow import flow
) to catch import/dependency issues early. 5) If you package code into the image instead of pulling at runtime - Ensure the ECS image contains your flow code at the exact path referenced by 
entrypoint
. Helpful references - Load flow behavior: load_flow_from_flow_run - prefect.yaml guidance: prefect.yaml how-to - ECS worker guide: ECS guide If you can share: - The output of 
prefect deployment inspect "flow/deployment"
- The relevant snippet of your 
prefect.yaml
(flows section with 
entrypoint
and 
pull
steps) - Whether the deployment points to a branch or a pinned commit I can help pinpoint exactly which of the above is biting you and propose a precise fix.
m
@Marvin Deployment inspect:
Copy code
{
    'id': '7edb2e42-5c12-42ba-acfa-a17f53a031e6',
    'created': '2025-11-04T17:05:21.805782+00:00',
    'updated': '2025-11-05T15:50:31.298315+00:00',
    'infra_overrides': {},
    'name': 'test-flow',
    'version': '48771cacc45c125eb6322346f0c484e0fc65f804',
    'description': None,
    'flow_id': '1e081348-2a38-4d51-be57-21c20f3698f4',
    'schedule': {'cron': '* * * * *', 'timezone': 'UTC', 'day_or': True},
    'is_schedule_active': True,
    'paused': False,
    'schedules': [
        {
            'id': '8d07a0c3-25db-40f4-aeda-37ea976f211c',
            'created': '2025-11-05T15:51:06.795815+00:00',
            'updated': '2025-11-05T18:19:48.001818+00:00',
            'deployment_id': '7edb2e42-5c12-42ba-acfa-a17f53a031e6',
            'schedule': {'cron': '* * * * *', 'timezone': 'UTC', 'day_or': True},
            'active': False,
            'max_active_runs': None,
            'max_scheduled_runs': None,
            'catchup': False
        }
    ],
    'parameters': {},
    'pull_steps': [{'prefect.deployments.steps.set_working_directory': {'directory': '/opt/prefect'}}],
    'tags': [' utils'],
    'work_queue_name': 'default',
    'last_polled': '2025-11-05T18:40:16.302958+00:00',
    'parameter_openapi_schema': {'type': 'object', 'title': 'Parameters', 'properties': {}},
    'path': None,
    'entrypoint': 'projects/test/flows.py:test_flow',
    'manifest_path': None,
    'storage_document_id': None,
    'infrastructure_document_id': None,
    'created_by': {'id': 'b141bd9c-233a-4f1f-835a-ae3f6f0d0c46', 'type': 'USER', 'display_value': 'upsilon26-mataline'},
    'updated_by': {'id': 'b141bd9c-233a-4f1f-835a-ae3f6f0d0c46', 'type': 'USER', 'display_value': 'upsilon26-mataline'},
    'work_queue_id': None,
    'enforce_parameter_schema': False,
    'work_pool_name': 'ascribe-ecs-worker-pool',
    'status': 'READY',
    'automations': []
}
prefect.yaml:
Copy code
# Generic metadata about this project
name: data-elt
prefect-version: 2.20.0

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect
The deployment points to an image that is updated in ecr on pushes to main branch
@Marvin
m
✅ thought for 153.4 seconds
Thanks for the details — this explains the behavior. What’s happening - Your deployment has no storage and no code “pull” (only a working directory change), and the entrypoint is set to 
projects/test/flows.py:test_flow
. - In Prefect 2.20.0, the worker will sometimes try to resolve/import the flow from the deployment entrypoint before submitting the run to ECS. Because the worker container itself does not have your project code at 
/opt/prefect/projects/test/flows.py
, the import fails and you get “Flow can not be retrieved from deployment.” Since that happens before submission, you never see an ECS task start. Why it’s “sometimes” - You update the ECR image on pushes to main. If the path or the function name changes across images, or the worker and the runtime image diverge, some runs will fail pre‑submission while others (built against an older image/layout) succeed. - Mutable image tags (e.g., “latest”) can race with scheduled runs; an entrypoint that used to be valid may briefly point at a moved/renamed file in the newly published image. What to change (pick one pattern and stick to it) - Option A: Pull code at runtime (recommended for 2.x) - Add a git clone pull step so the runtime environment always has code, and pin to a commit SHA to avoid drift:
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/your-org/your-repo.git>
        commit: 48771cacc45c125eb6322346f0c484e0fc65f804   # or a branch if you must
        access_token: "{{ prefect.blocks.github-credentials.your-token.access_token }}"
    - prefect.deployments.steps.set_working_directory:
        directory: your-repo
- Keep 
entrypoint: projects/test/flows.py:test_flow
relative to that repo root. - Docs: Pull steps API and Versioning deployments. - Option B: Bake the code into the ECS image and ensure the runtime can import it - Make sure the ECS task image contains the file at exactly 
/opt/prefect/projects/test/flows.py
and that the function is named 
test_flow
. - Use an immutable image tag (e.g., tag with the commit SHA) in your ECS work pool template to avoid “sometimes” mismatch. - If you stay on 2.20.0, runs can still fail pre‑submission if the worker tries to import and the worker container does not have the same code. To avoid that, either: - Run the worker from the same image that contains your project code, or - Move to Option A (git pull at runtime), or - Upgrade to Prefect 3.x where the worker uses a safer “placeholder flow” approach and does not hard‑fail on pre‑import in this pattern. - ECS guide: ECS worker guide. Quick checklist for your current setup - Confirm your task image actually contains 
/opt/prefect/projects/test/flows.py
and a function 
test_flow
. - Avoid mutable image tags; use one tag per build (ideally the commit SHA) and update the work pool to that tag. - If you prefer to keep a single, mutable image tag, add the git clone pull step so your container always has code that matches the entrypoint. If you can share these, I can pinpoint the exact fix: - 
prefect version
from the worker - The ECS work pool’s base job template (especially the image tag) - Whether the worker container image includes your project code at 
/opt/prefect/projects/...
- Any worker logs around the failed submission References - Flow loading behavior: load_flow_from_flow_run - Pull steps: Pull steps API - Versioning best practices: Versioning deployments
m
@Marvin please link the documentation on 2.20.0's pre-submission run failures
m
✅ thought for 218.9 seconds
Short answer: there isn’t a single 2.20.0 doc page titled “pre‑submission run failures.” In Prefect 2.x, the worker/agent loads your flow from the deployment entrypoint at run time; if the code isn’t available or the entrypoint is wrong, you’ll see “Flow could not be retrieved from deployment.” The behavior is described across these 2.20.0 docs: - Deployments (entrypoint is stored; flow is loaded at runtime): Deployments (2.20.0) - Storage (how code is made available to the runtime; without storage, your image/container must contain the code at the entrypoint path): Storage (2.20.0) - Work pools and workers (how workers poll/submit runs and interact with deployments): Work pools &amp; workers (2.20.0) Community issues showing the exact error in 2.x (pre‑submission retrieval failing due to import/entrypoint problems): - PrefectHQ/prefect#6700 — “Flow could not be retrieved from deployment” during run submission (2.3.1) - PrefectHQ/prefect#14363 — Import resolution leading to “Flow could not be retrieved from deployment” (2.19.6) If you’re looking for a specific claim or line in the docs and I missed it, let me know what wording you want verified and I’ll dig deeper.
m
@Marvin why would the code be unavailable sometimes and available other times
m
✅ thought for 62.9 seconds
Great question. In your setup the code is only “there” if the container that executes (or the worker doing a pre‑import) actually has the file at 
/opt/prefect/projects/test/flows.py
. Because you’re updating a single ECR tag on pushes to main and you don’t pull code at runtime, a few things can make it available sometimes and missing other times: Most common sources of intermittency - Mutable image tags: The same tag points to different images over time. Some runs start on the older image (where the path existed), others on the new image (path changed or code missing). - Worker vs runtime image mismatch (Prefect 2.20.0): The worker may attempt to load the flow before submitting the ECS task. If the worker container doesn’t include your project code, the pre‑import fails immediately. Depending on which worker handles a run (and which image it’s running), you’ll see “sometimes” failures. - Multiple workers or rolling updates: Different workers may run different images/versions, so some can import the entrypoint and others can’t. - Entry point drift: You changed the file layout or function name in the latest image, but the deployment’s 
entrypoint
still points to the old path. - Dependency drift: The new image is missing a dependency import used by the flow module; imports succeed on one image, fail on another. - Build variability: Docker context/.dockerignore occasionally omits the 
projects/
directory or copies it to a different path in some builds. Make it deterministic - Pin the runtime image per deploy: Tag each build with the commit SHA (or digest) and set your ECS work pool/template to that immutably tagged image. Update the pool when you release a new build. - Or pull code at runtime: Add a 
git_clone
pull step pinned to a commit SHA so the container always has the exact code the entrypoint expects. - Align worker and runtime images: If you stay on 2.20.0 without pull steps, run the worker with the same image that contains your code to avoid pre‑import failures. - Avoid path/function drift: Keep the 
entrypoint
in sync with your repo layout; don’t move/rename the flow without updating the deployment. Quick checks you can run - Verify the file exists in the image you think is running:
Copy code
docker run --rm <your-ecr-repo>:<tag> ls -la /opt/prefect/projects/test
docker run --rm <your-ecr-repo>:<tag> test -f /opt/prefect/projects/test/flows.py && echo "flows.py present"
- If you have multiple workers, confirm their images/tags and Prefect versions match. - Confirm your deployment’s 
entrypoint
matches the actual file/function in the image you’re using. Relevant docs - Deployments (2.20.0): Deployments - Storage and making code available at runtime: Storage - Work pools/workers: Work pools &amp; workers If you share the worker image tag, the ECS task image tag, and whether you have multiple workers, I can pinpoint which of the above is causing your “sometimes” and suggest the cleanest fix for your release flow.
m
@Marvin the worker image tag and the task image tag are both set to 'latest'. I only have the one worker in my work pool
m
✅ thought for 66.2 seconds
Thanks — using a mutable tag like ‘latest’ for both the worker and the ECS task image is the core reason you’re seeing “sometimes” behavior. Why “latest” causes intermittency - Tag drift at different points in time: The worker container is long‑lived and keeps whatever “latest” it started with; your ECS tasks resolve “latest” at submission time. When you push new images, the worker and the tasks can point at different code layouts. - Registry/agent caching and propagation: ECS/ECR can briefly serve/cached different digests for the same tag across nodes/regions. Two runs close together may land on different digests. - Code layout drift: If your repo moved/renamed 
projects/test/flows.py
(or the flow function) in a new build, some runs hit the old layout, others the new one, depending on which “latest” digest each component uses at that moment. - Dependency drift: New image drops/adds imports; pre‑import in the worker succeeds/fails depending on which “latest” it’s running. How to make it deterministic - Pin immutable images: - Build and push images tagged with the commit SHA (e.g., 
:48771cacc45c125e
) and reference that in: - Your ECS work pool base job template (task image) - The worker’s image - Update both when you release. Avoid 
:latest
for production flows. - Or pull code at runtime: - Add a 
git_clone
pull step pinned to the same commit SHA so the container always has the exact code the entrypoint expects:
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/your-org/your-repo.git>
        commit: 48771cacc45c125eb6322346f0c484e0fc65f804
        access_token: "{{ prefect.blocks.github-credentials.your-token.access_token }}"
    - prefect.deployments.steps.set_working_directory:
        directory: your-repo
- Keep 
entrypoint: projects/test/flows.py:test_flow
relative to that repo root. - Keep entrypoint and image in sync: - Don’t change the path or function name without updating the deployment. - Optionally, upgrade to Prefect 3.x: - 3.x improves the deployment workflow and avoids several 2.x pitfalls around code retrieval; you’d use 
flow.from_source(...).deploy(...)
and immutable sources instead of relying on mutable image tags. Helpful references - Deployments (2.20.0): Deployments (2.20.0) - Storage and making code available at runtime: Storage (2.20.0) - ECS worker guide: ECS guide If you share the exact ECS work pool base job template (image field) and how you deploy the worker (its image tag), I can suggest the minimal changes to switch from 
latest
to a commit‑pinned setup.
m
@Marvin I have both upgraded to prefect 3.x and made an immutable tag for my ecr image. Why does the flow not be able to be retrieved from deployment still?
m
✅ thought for 1254.2 seconds
Great question — in Prefect 3.x “Flow could not be retrieved from deployment” almost always means the worker can’t load your flow object at runtime. Upgrading to 3.x and using an immutable ECR tag are good steps, but there are a few other common culprits. Most frequent causes (and quick checks) - Entrypoint mismatch - The deployment’s 
entrypoint
must point to a file that actually exists in the runtime environment and a function/class that exists in that module, e.g. 
app/flows/my_flow.py:my_flow
. - Very common gotcha with containers: the file layout inside the image doesn’t match your entrypoint. For example, you 
COPY src/ /app/src
and 
WORKDIR /app
but your entrypoint is 
flows/my_flow.py:my_flow
instead of 
src/flows/my_flow.py:my_flow
. - Quick test inside the image:
Copy code
docker run --rm -it <your-ecr-image> sh -lc 'python -c "from prefect.flows import load_flow_from_entrypoint; f=load_flow_from_entrypoint(\"src/flows/my_flow.py:my_flow\"); print(\"Loaded:\", f.name)"'
Adjust the path/flow name until it loads without error. - Code not present at runtime - If you’re relying on your container image to carry the code, remove pull steps and make sure the code is baked into the image at the path used by 
entrypoint
. - If you’re relying on pull steps (e.g., clone from Git), verify they run successfully in the job environment (auth, commit exists, destination path). In 3.x, legacy storage blocks are gone; use 
flow.from_source(...)
or pull steps in your deployment config. - Docs: Store your flow code - Missing dependencies/import errors - If the module at your entrypoint imports packages that aren’t installed in the image, the flow won’t load. Make sure all runtime deps are installed in the image. - Image is not the one actually used at runtime - Even with “immutable tags” enabled in your ECR repo, if your deployment references a tag and your runtime environment does not pull a fresh image, you may be running a stale locally cached image. - For Kubernetes, set 
imagePullPolicy: Always
(via work pool job variables or your pod template). For Docker workers, ensure the worker pulls or the image isn’t cached. ECS generally pulls each time. - Consider pinning by digest in your deployment: 
<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/app@sha256:<digest|123456789012.dkr.ecr.us-east-1.amazonaws.com/app@sha256:<digest>>
- Docs: Deployment versioning and Customize job variables - ECR auth/permissions - Ensure the worker’s environment can pull from ECR (valid login on Docker worker host; correct IAM permissions and registry credentials for ECS/Kubernetes). Expired ECR logins can cause pull failures. How to quickly diagnose 1) Inspect the deployment (confirm entrypoint, pull steps, and image seen by Prefect):
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
2) Check worker logs for the flow run that failed — they usually reveal whether it’s a FileNotFoundError on the entrypoint, an ImportError, or a pull/auth issue. You can increase verbosity with 
PREFECT_LOGGING_LEVEL=DEBUG
in your job variables. 3) Validate the entrypoint inside your image locally (example above). If it fails locally, it will fail in production. 4) Confirm how you’re supplying code: - If using Python API:
Copy code
from prefect import flow

     @flow
     def my_flow():
         ...

     my_flow.from_source(
         source="<https://github.com/<org>/<repo>@<commit_sha>>",
         entrypoint="src/flows/my_flow.py:my_flow",
     ).deploy(
         name="prod",
         work_pool_name="<your-pool>",
         image="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/app@sha256:<digest>|123456789012.dkr.ecr.us-east-1.amazonaws.com/app@sha256:<digest>>",
     )
Docs: Flow.from_source
- Or ensure your image contains the code at the entrypoint path and omit pull steps. If you can share a bit more, I can pinpoint it fast: - Your work pool type (Docker, Kubernetes, ECS, etc.) - How you created the deployment (Python 
.deploy()
or 
prefect deploy
) and your 
entrypoint
- Whether you’re using pull steps or baking code into the image - The 
image
value stored in the deployment (digest vs tag) and any image pull policy/job variables - The exact error snippet from the worker logs Helpful references - Store flow code in Prefect 3.x - Versioning (pin commits/digests) - Customize job variables - Upgrade to workers (3.x vs 2.x) My hunch is an entrypoint/path mismatch inside the image or missing dependency. If you paste your 
prefect deployment inspect
output (redact secrets) and the worker error log, I’ll help you zero in on it.
m
@Marvin I am using an ECS type I'm baking code into the image Error log:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 936, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1073, in get_code
  File "<frozen importlib._bootstrap_external>", line 1130, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/opt/prefect/projects/test/flows.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 394, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 221, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/flows.py", line 973, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'opt/prefect/projects/test/flows.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
m
✅ thought for 28.3 seconds
Thanks for the logs — they point to an entrypoint path issue inside the ECS task container. Key clue: - Prefect is trying to load: 
/opt/prefect/opt/prefect/projects/test/flows.py
- The entrypoint being used appears to be 
opt/prefect/projects/test/flows.py:...
(note missing leading slash), which is being treated as relative and joined to the working dir 
/opt/prefect
, causing the duplicated path. How to fix - Make your entrypoint either: - Absolute: 
/opt/prefect/projects/test/flows.py:my_flow
, or - Relative to the container working directory (which is typically 
/opt/prefect
in Prefect’s images/templates): 
projects/test/flows.py:my_flow
- Ensure the file truly exists in the image at that path. Common pattern:
Copy code
# Dockerfile
  FROM prefecthq/prefect:3-python3.11
  WORKDIR /opt/prefect
  COPY . /opt/prefect
  # pip install -r requirements.txt etc.
Then use 
projects/test/flows.py:my_flow
as the entrypoint. Quick local verification (before re-deploy)
Copy code
docker run --rm -it <your-ecr-image> sh -lc 'ls -R /opt/prefect && python - <<PY
from prefect.flows import load_flow_from_entrypoint
f = load_flow_from_entrypoint("projects/test/flows.py:my_flow")
print("Loaded:", f.name)
PY'
Also double-check - Your deployment’s `entrypoint`:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Confirm it matches the path in the container (absolute or relative to 
/opt/prefect
). - Since you’re baking code into the image, remove any pull steps that might alter paths or create confusion. References - Store flow code (3.x): Store your flow code - Customize job variables (helpful for ECS settings): Customize job variables If you paste: - Your current deployment entrypoint string, and - The relevant Dockerfile COPY/WORKDIR lines, I can give you the exact entrypoint to use.
m
@Marvin I'm sorry. This is the log of an actual run, that was a test run with a wrong entrypoint: Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 936, in exec_module File "<frozen importlib._bootstrap_external>", line 1073, in get_code File "<frozen importlib._bootstrap_external>", line 1130, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/projects/test/flows.py' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 394, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 221, in load_flow_from_flow_run flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread return await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/flows.py", line 973, in load_flow_from_entrypoint flow = import_object(entrypoint) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 201, in import_object module = load_script_as_module(script_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module raise ScriptError(user_exc=exc, path=path) from exc prefect.exceptions.ScriptError: Script at 'projects/test/flows.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
m
✅ thought for 35.3 seconds
Got it — thanks for the corrected log. This error is now very specific: Prefect is trying to load a file-based entrypoint at projects/test/flows.py, which resolves to /opt/prefect/projects/test/flows.py inside the ECS task container, and that file does not exist in the image. That narrows it to one of these: - The file isn’t copied into the image (common: .dockerignore excluded it, or COPY path is wrong) - The file exists but at a different path (e.g., /app/projects/test/flows.py) while your entrypoint expects /opt/prefect/projects/test/flows.py - The deployment is not using your intended image (it’s using a base image or an older image without code) Quick checks to pinpoint the cause 1) Inspect the deployment (confirm entrypoint and image actually used)
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
- Confirm entrypoint shows something like 
projects/test/flows.py:my_flow
- Confirm job_variables.image is your ECR image (ideally pinned by digest) 2) Pull and test the exact image locally - Pull by the exact reference from the deployment (prefer digest):
Copy code
docker pull 123456789012.dkr.ecr.<region>.<http://amazonaws.com/your-repo@sha256:<digest|amazonaws.com/your-repo@sha256:<digest>>
- Check the file exists and can be loaded:
Copy code
docker run --rm -it 123456789012.dkr.ecr.<region>.<http://amazonaws.com/your-repo@sha256:<digest|amazonaws.com/your-repo@sha256:<digest>> \
       sh -lc 'ls -l /opt/prefect/projects/test || true; \
               python - <<PY
from prefect.flows import load_flow_from_entrypoint
f = load_flow_from_entrypoint("projects/test/flows.py:my_flow")
print("Loaded:", f.name)
PY'
- If ls shows “No such file or directory” or the Python load fails, the image doesn’t have the code at that path. 3) Align your Dockerfile, code path, and entrypoint - If you want to use a relative entrypoint like projects/test/flows.py:my_flow, use this pattern:
Copy code
FROM prefecthq/prefect:3-python3.11
     WORKDIR /opt/prefect
     COPY . /opt/prefect
     # RUN pip install -r requirements.txt
- If you copy to a different path (e.g., /app), either set WORKDIR /app and keep the relative entrypoint, or change the entrypoint to an absolute path, e.g. /app/projects/test/flows.py:my_flow. 4) Watch for .dockerignore - Ensure .dockerignore is not excluding projects/, test/, or .py files that contain your flow. 5) Make sure the deployment references the correct image - For ECS, the worker creates task defs from the deployment’s job variables. If your deployment does not set image, it may fall back to the work pool default image (which won’t have your code). - Best practice: reference your image by digest, not tag, e.g.:
Copy code
123456789012.dkr.ecr.<region>.<http://amazonaws.com/your-repo@sha256:<digest|amazonaws.com/your-repo@sha256:<digest>>
If you share: - Your deployment entrypoint string - The deployment’s job_variables.image (from inspect) - The relevant Dockerfile lines (WORKDIR and COPY) I can give you the exact entrypoint/path combo to use. References - Store flow code (3.x): Store your flow code - Customize job variables (ECS): Customize job variables
m
@Marvin does it change if the deployment works on occasion? It fails ~70% of the time but 30% of the time it works as intended
m
✅ thought for 74.7 seconds
Short answer: yes — intermittent success almost always means different runtime environments are being used across runs. In ECS that typically boils down to one of: - Some runs use your image with code; others use a base/pool-default image without your code - Pull steps are sometimes running (adding the files) and sometimes not - Different deployment versions or ECS task definition revisions are being invoked Given your error is “No such file or directory: /opt/prefect/projects/test/flows.py”, successful runs must be using an image that contains that file at that exact path; failed runs are using an image (or task definition) that does not. Checklist to pin this down 1) Inspect the deployment (single source of truth)
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
- EntryPoint should match the path in your image (e.g. 
projects/test/flows.py:my_flow
) - job_variables.image should be your ECR image, ideally pinned by digest (not a tag) - pull_steps should be empty if code is baked into the image 2) Verify the exact image each failing ECS task ran - In the ECS task details, confirm the image reference (it should match the digest from your deployment). If some tasks show a base 
prefecthq/prefect:...
or an older tag, that explains the misses. - If you see mixed images, set the image on the deployment and remove/override any image on the work pool default so the pool can’t override it. 3) Pull and test the image your deployment says it uses
Copy code
docker pull <your-ecr-repo>@sha256:<digest>
   docker run --rm -it <your-ecr-repo>@sha256:<digest> sh -lc 'ls -l /opt/prefect/projects/test || true; python - <<PY
from prefect.flows import load_flow_from_entrypoint
f = load_flow_from_entrypoint("projects/test/flows.py:my_flow")
print("Loaded:", f.name)
PY'
- If that fails locally, the image does not actually contain the file at that path (Dockerfile COPY/WORKDIR or .dockerignore issue). 4) Remove accidental pull steps - If you still have a git_clone or other pull step configured, it can make runs succeed only when the network/auth works. Since you bake code, remove them. 5) Check for multiple deployment versions being used - In the flow run details, compare deployment_id/version between a passing and failing run. If they differ, you’re running different deployment objects (with different images/entrypoints). If you can share: - The output of 
prefect deployment inspect "<flow>/<deployment>"
(entrypoint, job_variables.image, pull_steps) - Your ECS work pool default job variables (especially any “image” there) - The image shown on a failing ECS task - The relevant Dockerfile COPY/WORKDIR lines …I’ll help you zero in on the exact mismatch and provide the precise fix. Helpful refs: - Store flow code (3.x): Store your flow code - Deployment versioning: Versioning - Customize job variables (ECS): Customize job variables
Open in Slack
PreviousNext
Powered by