trying to submit a deployment with ECS task overrides as json patches.....here is the command

prefect deployment build example_flow.py:flow_1 -n base -sb s3/common-utils-dev-test/flow-group-1/flow-1 -ib ecs-task/common-utils-flow-1-dev-test --override 'task_customizations=[{"op": "replace", "path": "/overrides/cpu", "value": "512"}, {"op": "replace", "path": "/overrides/memory", "value": "1024"}]' -q prefect-v2-queue-dev-test -v dev --params '{"param_name": "param_value"}' -t common-utils -t flow-group-1 -a --cron '0 0 * * *'

It results in this infra override in deployments

{
  "task_customizations": "[{\"op\": \"replace\", \"path\": \"/overrides/cpu\", \"value\": \"512\"}, {\"op\": \"replace\", \"path\": \"/overrides/memory\", \"value\": \"1024\"}]"
}

however when I submit the flow run I get this error

Submission failed. pydantic.error_wrappers.ValidationError: 1 validation error for ECSTask task_customizations instance of JsonPatch expected (type=type_error.arbitrary_type; expected_arbitrary_type=JsonPatch)

anyone know why this would not work.

Cannot get this test to pass Flake8....any suggestions. noqua is not working.

@pytest.mark.usefixtures("ecs_mocks")
async def test_task_customizations_as_string(aws_credentials):

    task = ECSTask(
        aws_credentials=aws_credentials,
        memory=512,
        cpu=256,
        task_customizations='[{"op": "replace", "path": "/overrides/cpu", "value": "512"}, {"op": "replace", "path": "/overrides/memory", "value": "1024"}]',
    )  # type: ignore

    original_run_task = task._run_task
    mock_run_task = MagicMock(side_effect=original_run_task)
    task._run_task = mock_run_task

    await run_then_stop_task(task)

    overrides = mock_run_task.call_args[0][1].get("overrides")

    assert overrides["memory"] == "1024"
    assert overrides["cpu"] == "512"

I am trying to install prefect on ec2 (Amazon Linux) but running into the asyncpg related error. Does anyone have any solution for that

Hi all, I'm hoping someone can assist in solving this issue I'm experiencing when running a Prefect Agent on ECS. It seems that the ECS Task works fine when it's started by the ECS Service and will pick up any work that has been scheduled or triggered. However, after a couple of hours of the ECS Task waiting for work, it throws an error (see attached screenshots of Cloudwatch Logs). After this error is thrown, the ECS Task container stays active but does not pick up any work. This causes schedule flows to be late until I manually stop the container and let the ECS Service provision a new ECS Task container. This is not ideal as we have some flows scheduled for graveyard hours. Two questions: 1. Any idea what is causing this error? 2. Is there a configuration that I am missing that will stop the container when an error occurs so that the ECS Service can provision a fresh container?

has anybody else ever encountered this error? (

ListObjects: permission denied

) https://github.com/PrefectHQ/prefect/issues/9188 it's quite a painful one since it appears to happen very randomly and i have no idea how to get around it

happens when pulling flows down from s3

anyone run into issues starting prefect as a service - I can run prefect server start when logged into the Ec2 and it works fine and reads my profiles.toml but, when running under systemctl I get and Application startup failed Error saying that app.state does not have attribute services. Here is my service definition: [Unit] Description=Prefect Server After=network.target [Service] User=prefect Group=prefect WorkingDirectory=/home/prefect ExecStart=/usr/local/bin/prefect server start Restart=always [Install] WantedBy=multi-user.target

I have a flow running with ECS Tasks as subflows, and noticing I cannot Cancel any of my flows. Is this a known issue?

Hi, i'm just wondering, if it is not possible to give your ECS Task the possibility to specify how many gpus this task should use below an EC2 instance. Or is it not necessary? Does someone have an example, how I can run an ECS Task with GPU utilization? https://prefecthq.github.io/prefect-aws/ecs/ Below this documentation, I can't find it.

Hi, what is the easiest way to start a lambda function when a flow-run reaches a Completed state? I tried with Automations but it is not automatically built in. Optimally there would be a way to run a small python script after a flow run reaches a specific state. If that is possible, even with a workaround, it would be great

Hello everyone, I’m currently setting up a deployment that includes an ECSTask infrastructure block and an S3 storage block. To customize the ECSTask, I’m using a custom docker image with the dockerfile below So, my question is: What is missing from my Dockerfile image or Deployment object so it retrieves the flow correctly from the downloaded files in the S3 storage? Thank you in advance for any help you can provide.

it looks like sometimes the worker process loses connection to the server, does anyone else have this problem? I'm using the chart here https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-worker/Chart.yaml. the work pool becomes unhealthy every few hours without warning and I have to kick the worker

Hi, I have a pipeline in AWS where things trigger each other and would like to move it entirely to prefect but have some doubts/questions: 1. There is a Lambda_1 triggered on a schedule via cron job in Eventbridge. At the end this Lambda_1 has some python code to trigger Lambda_2 a. Q1.1) Can I trigger Lambda_1 in a Prefect deployment? b. Q1.2) Can Lamba_2 be triggered as well from Prefect depending on the completion status of Lambda_1? 2. There is a Lambda_2 that at the end of its python code does the following: updates the Fargate service for Container_1 to desired count = 1 (basically starts a task that wil be billed starting then). a. Q2.1) Can I update, with Prefect, the service for Container_1 to have 1 task running, depending on the completion status of Lambda_2? b. Q2.2) Can I start, with Prefect, the Prefect deployment for Container_1 depending on the completion status of the service update? c. Q2.3) Can I, with Prefect, update the service again to 0 so that it stops the billing? this should also depend on the completion status. If all python code was succesfull then stop and if not then retry a couple of times the flow for the deployment in Container_1 3. The Container_1 has a Prefect deployment that at the end calls Lambda_3 and Lambda_4 4. Lambda_3 is similar to Lambda_4 and it starts updates another fargate service to have 1 task running. Then it should start the Prefect deployment inside Container_2 5. Container_2 is similar to Container_1. It should call another lambda if the python code run succesfully and should retry a couple of times if there was any errors. At the moment I have all these triggers/calls within the codes and have to no proper way to see unified logs for everything so I thought Prefect could help and simplify my experience. General suggestions on the overall architecture are also welcome :)Thanks!

I get this error when the flow runs. I have not changed anything from my setup

Submission failed. botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the RegisterTaskDefinition operation: The security token included in the request is invalid

hey, I updated the prefect kube charts today, and my job has now failed with

botocore.exceptions.SSLError: SSL validation failed for <https://my-bucket.s3.amazonaws.com/native/my/path/to/file.json> EOF occurred in violation of protocol (_ssl.c:2423)

has anybody encountered this? is this likely just a permissions / role problem?

ECSTask

won't be deprecated until Workers & Projects are out of Beta, right?

When I cancel the workflow in Prefect cloud, it is not stopping the task in ECS, Is there a way to do that, probably in ECS task block configuration?

Any chance to get a new release for the

prefect-aws

package to fix this issue: https://github.com/PrefectHQ/prefect-aws/pull/273?

FYI, not Prefect related but in case anyone is using it: AWS is experiencing an issue with Lambdas on US-EAST-1.

Hello all! I'm working on getting the secret block up and running. The issue I'm running into is that the secret is saved as a "SecretString" in aws. When I pull it out of the block, the block is looking for

SecretBinary

and reading the block like this:

from prefect_aws import AwsSecret

def my_flow():
    secrets_manager = AwsSecret.load("aaron-secret")
    print(secrets_manager.read_secret())

Gives me a

KeyError: 'SecretBinary'

Hi, we have suddenly getting these errors in our cluster

ImportError: cannot import name 'SecretField' from 'pydantic' (/usr/local/lib/python3.8/dist-packages/pydantic/__init__.py)

all prefect job pods are in Error stage

Hey! Curious if anyone else has run into this. I have my prefect2 flows running on ECS tasks with s3 storage. We got a security warning that Amazon ECS containers should have read-only access to its root filesystems that we need to address for our security protocols. I added

"readonlyRootFilesystem": true,

to the container definition, but then the flow fails to load. Looks like the error is when it tries to make the directory and can't, which makes sense

File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/opt/prefect/./<flow_name>

Hi, We are intermittently getting this error in some of our prefect flows running on 2.8.4 and tried increasing resources as well but nothing works. If someone has faced this earlier, please help. Setup is running on kubernetes.

AssertionError

Traceback (most recent call last):

  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)

  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()

  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()

  File "/usr/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)

  File "/usr/lib/python3.8/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)

  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 1670, in cancel_flow_run
    raise TerminationSignal(signal=signal.SIGTERM)

prefect.exceptions.TerminationSignal


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 1533, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)

  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 165, in run_sync_in_interruptible_worker_thread
    assert result is not NotSet

AssertionError

Finished in state Failed('Task run encountered an exception: Traceback (most recent call last):\n  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run\n    return loop.run_until_complete(main)\n  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete\n    self.run_forever()\n  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever\n    self._run_once()\n  File "/usr/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once\n    event_list = self._selector.select(timeout)\n  File "/usr/lib/python3.8/selectors.py", line 468, in select\n    fd_event_list = self._selector.poll(timeout, max_ev)\n  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 1670, in cancel_flow_run\n    raise TerminationSignal(signal=signal.SIGTERM)\nprefect.exceptions.TerminationSignal\n\nDuring handling of the above exception, another exception occurred:\n\nAssertionError\n')

is it possible for prefect to get triggered by a event webhook coming from a lambda function?

Hi, I setup an ECS Push Work Pool on Prefect Cloud using an Auto Scaling group capacity provider with a MinSize of zero. On attempting to run a simple hello world flow, I saw the following message which leads me to believe perhaps I shouldn't expect the flow to wait for auto scaling to kick in before bailing out. Is that the case or am I missing something?

Flow run could not be submitted to infrastructure: Failed to run ECS task, cluster 'arn:aws:ecs:******' does not appear to have any container instances associated with it. Confirm that you have EC2 container instances available.

Hey y'all, has anyone experienced this? If the ecs job is pending for more than 120 seconds, I get a crashed alert when I go to the UI, and I see the flow started 2m18s late, yet the flow is completed. How can I extend this timeout?

RuntimeError: Timed out after 120.82966995239258s while watching task for status RUNNING

Hello everyone, I'm trying to inject aws credentials as env variables into my ecs tasks using an ecs work pool env variables override. I'm collecting the credentials from an existing aws credentials block like so :

{
  "AWS_ACCESS_KEY_ID": "{{ prefect.blocks.aws-credentials.<my-aws-block>.aws_access_key_id }}",
  "AWS_SECRET_ACCESS_KEY": "{{ prefect.blocks.aws-credentials.<my-aws-block>.aws_secret_access_key }}"
}

Unfortunately, my ecs tasks crashes at submission time with this error :

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 831, in _submit_run_and_capture_errors
    configuration = await self._get_configuration(flow_run)
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 906, in _get_configuration
    configuration = await self.job_configuration.from_template_and_values(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 125, in from_template_and_values
    populated_configuration = await resolve_block_document_references(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/templating.py", line 205, in resolve_block_document_references
    updated_value = await resolve_block_document_references(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/templating.py", line 205, in resolve_block_document_references
    updated_value = await resolve_block_document_references(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/templating.py", line 228, in resolve_block_document_references
    block_type_slug, block_document_name = (
ValueError: too many values to unpack (expected 2)

I guess it has to do with the formatting, but I could not find the related documentation. Any help would be appreciated!

Hi 👋 We are trying to transition from regular ECS blocks to the new Push Worker Pools.

Hi @Jake Kaplan I just forked the

prefect-aws

repo and am creating a venv so I can start work on https://github.com/PrefectHQ/prefect-aws/issues/310. Do you have a recommended python version?