<@ULVA73B9P>
Failed to submit flow run '7f9149f9-c...
# ask-marvinm
Madushan Pathirana
08/20/2025, 10:36 AM@Marvin
Failed to submit flow run '7f9149f9-ceeb-4a5f-be77-dd279491475d' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/docker/api/client.py", line 275, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 1026, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.47/images/create?tag=2025-08-19t04-28-44-524659-00-00&fromImage=010614160260.dkr.ecr.ap-southeast-1.amazonaws.com%2Fcredilinq-prefect
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 1262, in _submit_run_and_capture_errors
result = await self.run(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/prefect_docker/worker.py", line 457, in run
container, created_event = await run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
result = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2476, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark
return call()
^^^^^^
File "/usr/local/lib/python3.12/site-packages/prefect_docker/worker.py", line 727, in _create_and_start_container
self._pull_image(docker_client, configuration)
File "/usr/local/lib/python3.12/site-packages/prefect_docker/worker.py", line 876, in _pull_image
return docker_client.images.pull(image, tag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/docker/models/images.py", line 464, in pull
pull_log = self.client.api.pull(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/docker/api/image.py", line 429, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.12/site-packages/docker/api/client.py", line 277, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2025-08-19t04-28-44-524659-00-00&fromImage=010614160260.dkr.ecr.ap-southeast-1.amazonaws.com%2Fcredilinq-prefect: Internal Server Error ("Head "https://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/v2/credilinq-prefect/manifests/2025-08-19t04-28-44-524659-00-00": no basic auth credentials")
040323 PM
prefect.flow_runs.worker
Reported flow run '7f9149f9-ceeb-4a5f-be77-dd279491475d' as crashed: Flow run could not be submitted to infrastructure:
APIError(HTTPError('500 Server Error: Internal Server Error for url: http+docker://localhost/v1.47/images/create?tag=2025-08-19t04-28-44-524659-00-00&fromImage=010614160260.dkr.ecr.ap-southeast-1.amazonaws.com%2Fcredilinq-prefect'))
m
Marvin
08/20/2025, 10:36 AM✅ thought for 224.8 seconds
Marvin
08/20/2025, 10:40 AMShort version: your Docker worker couldn’t authenticate to your private ECR, so the image pull failed. Prefect is just the messenger here; fix auth on the machine running the Docker worker.
What happened
- Docker tried to pull 010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:2025-08-19t04-28-44-524659-00-00
- Docker responded “no basic auth credentials” → not logged in to that ECR registry (or token expired)
How to fix (pick one auth method)
Option A: One-time docker login (must refresh every 12h)
1) Ensure the worker host has AWS CLI and credentials with these ECR permissions: ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer (scoped to the repo/region).
2) Login to ECR:
Copy code
aws ecr get-login-password --region ap-southeast-1 \
| docker login --username AWS --password-stdin <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com> Copy code
docker pull <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:2025-08-19t04-28-44-524659-00-00|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:2025-08-19t04-28-44-524659-00-00> Copy code
cat ~/.docker/config.json
{
"credHelpers": {
"<http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com>": "ecr-login"
}
} Copy code
# /etc/systemd/system/prefect-worker.service.d/override.conf
[Service]
Environment="AWS_REGION=ap-southeast-1"
# Or provide keys (prefer roles over static keys)
# Environment="AWS_ACCESS_KEY_ID=..."
# Environment="AWS_SECRET_ACCESS_KEY=..."
# Environment="AWS_SESSION_TOKEN=..."systemctl daemon-reload && systemctl restart prefect-worker Copy code
aws ecr describe-images \
--region ap-southeast-1 \
--repository-name credilinq-prefect \
--image-ids imageTag=2025-08-19t04-28-44-524659-00-00image_pull_policy