<@ULVA73B9P> Task runs went missing when self-host...
# ask-marvin
f
@Marvin Task runs went missing when self-host the prefect in ECS. When I run the flow locally (in local server) I can see the tasks. How to fix it?
m
thought for 350.4 seconds
Got it — when task runs don’t show up only on your self‑hosted ECS server (but do locally), it’s almost always one of these: - The server loop services that record task runs aren’t running in ECS - The UI or flows are pointed at the wrong API URL - The server/database version/migrations are out of sync Assuming you’re on Prefect 3.x; if you’re on 2.x let me know and I’ll adjust. Quick checks to run inside your ECS server container - Verify Prefect and that services are enabled
Copy code
prefect version
  prefect server services list-services
You should see services like scheduler, event_persister, event_logger, and task_run_recorder. If they’re disabled or not listed, your server is likely running without services. - If you started the API without services (e.g., via uvicorn or with --no-services), either: - Start the monolithic server (web + services):
Copy code
prefect server start
Make sure you are NOT using the
--no-services
flag. - Or, if you must run the API separately, start the loop services in a separate process/sidecar:
Copy code
prefect server services start
Critical environment variables - Ensure these are set in your server container (they’re enabled by default, but can be disabled by env): -
PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED=true
-
PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED=true
-
PREFECT_API_SERVICES_EVENT_LOGGER_ENABLED=true
- If you’re toggling others, also keep
PREFECT_API_SERVICES_SCHEDULER_ENABLED=true
- Ensure the UI points to the correct API: -
PREFECT_UI_API_URL=<https://your-api-domain>
(whatever your ECS API is) - Ensure flows/workers can report to your API: -
PREFECT_API_URL=<https://your-api-domain>
inside your worker and flow containers Database migrations - If you upgraded Prefect in ECS, run the DB migrations once:
Copy code
prefect server database upgrade
A schema mismatch can prevent new entities from being recorded cleanly. What I’d like to confirm - Do flow runs appear in the UI and only the task runs are missing, or are both missing? - How are you starting the server in ECS (e.g.,
prefect server start
, a custom
uvicorn
command, or with
--no-services
)? - What are the values of
PREFECT_API_URL
(in your worker/flow containers) and
PREFECT_UI_API_URL
(in your UI container)? - Output of
prefect server services list-services
from the ECS server container References - Prefect 3 settings reference (includes server/UI/services env vars): Settings reference Once you share those details, I can help pinpoint exactly which toggle or process needs to be fixed.