https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> - running into this error - nothing h...
# marvin-ai
d
@Marvin - running into this error - nothing has changed in terms of my environment
Copy code
Failed to submit flow run '0147a3d2-e189-487f-9434-d3905f69b4bc' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 1011, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect_aws/workers/ecs_worker.py", line 680, in run
    ) = await run_sync_in_worker_thread(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect_aws/workers/ecs_worker.py", line 762, in _create_task_and_wait_for_start
    template_with_placeholders = self.work_pool.base_job_template[
                                 ^^^^^^^^^^^^^^
AttributeError: 'ECSWorker' object has no attribute 'work_pool'. Did you mean: '_work_pool'?
m
The error you're encountering seems to be related to the 
ECSWorker
object not having a 
work_pool
attribute. This could be due to a mismatch or misconfiguration in the Prefect setup, particularly with how work pools are being defined or utilized in your environment. Here's what you can do to address this: 1. Verify Work Pool Configuration: Ensure that your work pool is correctly configured and that the 
base_job_template
is properly set. You might want to create or update your work pool configuration. Here's an example snippet from a script that shows how to create a work pool:
Copy code
python
   from prefect import flow, get_client
   from prefect.client.schemas.actions import WorkPoolCreate

   async def create_work_pool():
       async with get_client() as client:
           await client.create_work_pool(
               WorkPoolCreate(
                   name="ecs-work-pool",
                   description="A work pool for ECS work",
                   base_job_template=dict(
                       variables=dict(
                           type="object", properties=dict(vpc_id=dict(type="string"))
                       ),
                       job_configuration=dict(
                           vpc_id="{{ prefect.variables.some_vpc_id }}"
                       ),
                   ),
               )
           )
2. Review ECS Worker Documentation: Make sure to review the ECS Worker Guide to ensure that your ECS setup aligns with Prefect's expectations. If you haven't changed anything in your environment, consider whether there might have been updates or changes in the Prefect library or its dependencies that could affect the behavior. If the issue persists, it might be worth checking for any recent updates or patches in the Prefect library that could address this problem.
j
Hey I think 
prefect-aws
likely needs a higher prefect compatible version. You can either downgrade 
prefect-aws
to 
0.5.6
or upgrade 
prefect>=3.2.13
to fix
We'll release a 
prefect-aws==0.5.8
with the correctly pinned prefect dependency shortly
n
prefect-aws==0.5.8
is now released - sorry about that!
d
thanks! just to clarify this is the worker or the image for the flow?
have been using the following with a consistent image of the worker for quite a while with no issues
Copy code
dependencies = [
    "prefect-docker>=0.6.2",
    "prefect[aws,github]>=3.2.7",
]
Bumped up to 
prefect[aws,github]>=3.2.13
but still seeing the same error..
n
can you try using the following on your worker?
Copy code
prefect[github]>=3.2.13
prefect-aws>=0.5.8
depending on how you're running your worker, this 
prefect[aws,github]>=3.2.13
doesn't necessarily guarantee you're getting the newest 
prefect-aws
d
yeah was giving that a shot next - will let you know shortly
👍 1
Copy code
dependencies = [
    "prefect-aws>=0.5.8",
    "prefect>=3.2.14",
    "prefect-docker>=0.6.2",
]
Same error @Nate
n
can you confirm that you're actually ending up with the expected versions?
d
yeah sure - one moment
@Nate details below The task definition that is being executed by the workpool for the target flow
The task definition image reference
The pyproject.toml when execing into the container of that image
Screenshot 2025-03-25 at 3.24.21 PM.png
The error
Copy code
Failed to submit flow run 'e5341de9-f047-4730-b45a-05b3fa81fe91' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/workers/base.py", line 1011, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect_aws/workers/ecs_worker.py", line 680, in run
    ) = await run_sync_in_worker_thread(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect_aws/workers/ecs_worker.py", line 762, in _create_task_and_wait_for_start
    template_with_placeholders = self.work_pool.base_job_template[
                                 ^^^^^^^^^^^^^^
AttributeError: 'ECSWorker' object has no attribute 'work_pool'. Did you mean: '_work_pool'?
n
the requested dep values are one thing, but if you have EXTRA_PIP_PACKAGES or some other thing going on, you can end up with different versions
d
Apologies for inundating. I'll continue to dive into it. Nothing has changed in my setup between yesterday and today and things were working fine.
n
can you run 
pip list | grep prefect
or similar while exec'd in?
d
Screenshot 2025-03-25 at 3.26.39 PM.png
oh shoot i see - error between the keyboard and chair
my dockerfile
Copy code
FROM prefecthq/prefect:3.1.5-python3.10 AS base

COPY --from=<http://ghcr.io/astral-sh/uv:0.5.9|ghcr.io/astral-sh/uv:0.5.9> /uv /uvx /bin/

ARG WORKDIR=/opt

WORKDIR $WORKDIR

RUN apt-get update && \
    apt-get install -y --no-install-recommends make && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY . ./

ENV PATH="/app/.venv/bin:$PATH"
ENV UV_PROJECT_ENVIRONMENT=/opt/.venv

RUN --mount=type=secret,id=codeartifact_token \
    export UV_INDEX_PRIVATE_PYPI_PASSWORD="$(cat /run/secrets/codeartifact_token)" && \
    export UV_INDEX_ADONIS_ML_MONO_PASSWORD="$(cat /run/secrets/codeartifact_token)" && \
    uv sync
honestly still fighting this. the error tells me prefect is installed, made all of the appropriate changes. but the log seems like it may be the worker, not the task runner for the flow being triggered?
n
wait sorry when you exec'd in and checked the deps, was that the container that runs your worker? or the flow run bc for something like an ECS worker (or k8s) there are 2 task definitions (i.e. images), 1 for the worker, one for the flow run usually the task definition for the worker service just needs 
prefect
(or 
prefect-aws
here because ECS worker) and then you can set 
image
on the work pool and we'll create a task definition automatically (unless you provide your own otherwise on the work pool) that uses that 
image
for flow run containers sorry if I'm saying things you already know, i'm just trying to figure out what the disconnect is
👍 1
for example, i've seen people inadvertently set this field (task definition arn for the flow run container) to their worker's task definition, which doesn't actually have their flow run deps (bc it only has the deps to run the worker)
d
I have two task definitions, one for the worker and another for the flow run. The requirements I shared with you were updates I applied to the run (failed). I realized I needed to update the task worker. I pinned prefect-aws 0.5.5 to the worker and that worked. When I updated to the latest version that failed again.
Something about this update appears to break the worker between 0.5.5 and latest for the worker (i.e. ECSWorker not having 
self.work_pool
). This makes sense because yesterday, as is, my worker was working without issues. Today with the updates it no longer is (I was never pinning a version to the worker). https://github.com/PrefectHQ/prefect/commit/e33dab815ad01b7ba611910500fd3fafc9a22d72 I'm waiting for a PR review to test the following command for my updated worker. Anticipating is will fail.
Copy code
"pip install \"prefect[aws]>=3.2.13\" && prefect worker start --pool ${var.prefect.work_pool.name} --type ${var.prefect.work_pool.kind}"
I'm designing my worker off of the following docs. https://docs.prefect.io/integrations/prefect-aws/ecs_guide#step-3%3A-creating-an-ecs-worker-service
@Nate Updates from testign worker versions: • prefect-aws>=0.5.6: does not work (results in error I shared) • prefect-aws=0.5.5: schedules tasks successfully • prefect[aws]>=3.2.13: schedules tasks successfully
35 Views
Open in Slack
PreviousNext
Powered by