Hi All, I'm on Prefect 3 latest. I'm trying to dep...
# ask-community
c
Hi All, I'm on Prefect 3 latest. I'm trying to deploy to ECS. I have my worker container already running in ECS and looks to be functioning properly. I have my deploy code in a ``my_flow_deploy.py` file, eample:
Copy code
from prefect import flow
from prefect.logging import get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    logger.info("Hello from ECS!!")


if __name__ == "__main__":
    my_flow()

def main():

    # Define the Deployment (No `ECSTask` needed in Prefect 3)
    my_flow.deploy(
        name="test-ecs-deployment",
        work_pool_name=WORK_POOL_NAME,
        parameters={},
        description="Prefect 3 deployment for test ECS flow",
        version="1.0.0",
        image=DockerImage(
            name=f"{ECR_REPO}",  # Must be this and not prefect otherwise we can't push!
            dockerfile="Dockerfile.flow",
        ),
        job_variables={  # ✅ Overrides ECS settings per deployment
            # "image": ECR_REPO,  # Image to use
            "cpu": 512,  # CPU Allocation
            "memory": 1024,  # Memory Allocation
            "cluster": CLUSTER_NAME,  # ECS Cluster Name
            "task_role_arn": TASK_ROLE_ARN,  # IAM Task Role
            "execution_role_arn": EXECUTION_ROLE_ARN,  # IAM Execution Role
            "networkConfiguration": {  # ✅ Ensures correct networking
                "awsvpcConfiguration": {
                    "subnets": SUBNETS,
                    "securityGroups": SECURITY_GROUPS,
                    "assignPublicIp": "ENABLED",
                }
            },
            "env": {  # Environment Variables (Secrets from AWS SSM & Secrets Manager)
                "PREFECT_API_URL": PREFECT_API_URL,
                "PREFECT_API_KEY": PREFECT_API_KEY,
            },
        },
        push=True,
        build=True,
    )
My Dockerfile.flow looks like this:
Copy code
FROM prefecthq/prefect:3-latest

# Set the working directory
WORKDIR /opt/prefect

# Copy requirements.txt first to leverage Docker caching
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the all code from local repo including .env
COPY . /opt/prefect

# Export environment variables automatically (optional)
ENV $(cat /opt/prefect/.env | xargs)
I build and deploy:
python my_flow_deploy.py
and it seems to build and deploy to both Prefect CLoud and the image to my ECR:
Copy code
$ python deployments/my_flow_deploy.py
14:43:25.459 | DEBUG   | prefect.profiles - Using profile 'cloud-dmp-dev'
14:43:25.466 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - Running call coroutine_wrapper() in thread 'RunSyncEventLoopThread'
14:43:25.466 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='coroutine_wrapper' RUNNING, runtime=0.00> entered
14:43:25.466 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='coroutine_wrapper' COMPLETED, runtime=0.00> exited
14:43:25.466 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - Scheduling coroutine for call coroutine_wrapper(<dropped>) in running loop <_UnixSelectorEventLoop running=True closed=False debug=False>
14:43:25.466 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - <AsyncCancelScope, name='coroutine_wrapper' RUNNING, runtime=0.00> entered
14:43:25.478 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspaces/4fd6069e-1234-abcd-b6b0-abcd1234abcd/>
14:43:26.108 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspaces/4fd6069e-1234-abcd-b6b0-abcd1234abcd/>
Successfully built image '00000000000.dkr.ecr.ap-southeast-1.amazonaws.com/my-repo:latest'
Successfully pushed image '00000000000.dkr.ecr.ap-southeast-1.amazonaws.com/my-repo:latest'
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--14:43:38.429 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspaces/4fd6069e-1234-abcd-b6b0-abcd1234abcd/>
Successfully created/updated all deployments!

                    Deployments                    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Name                        ┃ Status  ┃ Details ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ my-flow/test-ecs-deployment │ applied │         │
└─────────────────────────────┴─────────┴─────────┘

To schedule a run for this deployment, use the following command:

        $ prefect deployment run 'my-flow/test-ecs-deployment'


You can also run your flow via the Prefect UI: <https://app.prefect.cloud/account/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspace/4fd6069e-1234-abcd-b6b0-abcd1234abcd/deployments/deployment/a358659e-9529-420f-826d-38d45c3bf4f2>

14:43:39.533 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - <AsyncCancelScope, name='coroutine_wrapper' COMPLETED, runtime=14.07> exited
14:43:39.533 | DEBUG   | RunSyncEventLoopThread | prefect._internal.concurrency - Finished async call coroutine_wrapper(<dropped>)
I then run it from the cli (can also do it from the UI)
$ prefect deployment run 'my-flow/test-ecs-deployment'
Now, I see from ECS a new task runs, the container starts, then it fails with this error message:
exec /usr/bin/tini: exec format error
I've been struggling with this for quite some time - Marvin has helped a little but, but... still struggling. (Also, another issue, I see quite frequently it runs the container in mu private subnets - the subnets I ecplicitely set in
SUBNETS
are my public ones since I need internet access and I don't want to have to mess with NAT Gateways right now.) I largely followed the https://docs.prefect.io/integrations/prefect-aws/ecs_guide#1-write-a-simple-test-flow documentation but the last bit about actual deplpyments doesn't go into a whole lot of detail.. And no, I'm not using a prefect.toml - I opted to have all flow deployment specific config in the xxx_deploy.py files. Is this an ok decision to make? Thanks so much! (I'm a mac "M" user btw, if that helps)
e
Not a worker for prefect but that exec format error is because you are using an arm64 image instead of amd64 version or vice versa
c
Thanks @Emilio, yeah, I ended up going the prefect.yaml route. I got past the architecture issue using
platform: linux/amd64
in
prefect_docker.deployments.steps.build_docker_image:
. Now running into some other issues. making step by step progress!