Chris Drumgoole
02/10/2025, 6:51 AMfrom prefect import flow
from prefect.logging import get_run_logger
@flow
def my_flow():
logger = get_run_logger()
logger.info("Hello from ECS!!")
if __name__ == "__main__":
my_flow()
def main():
# Define the Deployment (No `ECSTask` needed in Prefect 3)
my_flow.deploy(
name="test-ecs-deployment",
work_pool_name=WORK_POOL_NAME,
parameters={},
description="Prefect 3 deployment for test ECS flow",
version="1.0.0",
image=DockerImage(
name=f"{ECR_REPO}", # Must be this and not prefect otherwise we can't push!
dockerfile="Dockerfile.flow",
),
job_variables={ # ✅ Overrides ECS settings per deployment
# "image": ECR_REPO, # Image to use
"cpu": 512, # CPU Allocation
"memory": 1024, # Memory Allocation
"cluster": CLUSTER_NAME, # ECS Cluster Name
"task_role_arn": TASK_ROLE_ARN, # IAM Task Role
"execution_role_arn": EXECUTION_ROLE_ARN, # IAM Execution Role
"networkConfiguration": { # ✅ Ensures correct networking
"awsvpcConfiguration": {
"subnets": SUBNETS,
"securityGroups": SECURITY_GROUPS,
"assignPublicIp": "ENABLED",
}
},
"env": { # Environment Variables (Secrets from AWS SSM & Secrets Manager)
"PREFECT_API_URL": PREFECT_API_URL,
"PREFECT_API_KEY": PREFECT_API_KEY,
},
},
push=True,
build=True,
)
My Dockerfile.flow looks like this:
FROM prefecthq/prefect:3-latest
# Set the working directory
WORKDIR /opt/prefect
# Copy requirements.txt first to leverage Docker caching
COPY requirements.txt .
# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy the all code from local repo including .env
COPY . /opt/prefect
# Export environment variables automatically (optional)
ENV $(cat /opt/prefect/.env | xargs)
I build and deploy:
python my_flow_deploy.py
and it seems to build and deploy to both Prefect CLoud and the image to my ECR:
$ python deployments/my_flow_deploy.py
14:43:25.459 | DEBUG | prefect.profiles - Using profile 'cloud-dmp-dev'
14:43:25.466 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - Running call coroutine_wrapper() in thread 'RunSyncEventLoopThread'
14:43:25.466 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='coroutine_wrapper' RUNNING, runtime=0.00> entered
14:43:25.466 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - <WatcherThreadCancelScope, name='coroutine_wrapper' COMPLETED, runtime=0.00> exited
14:43:25.466 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - Scheduling coroutine for call coroutine_wrapper(<dropped>) in running loop <_UnixSelectorEventLoop running=True closed=False debug=False>
14:43:25.466 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - <AsyncCancelScope, name='coroutine_wrapper' RUNNING, runtime=0.00> entered
14:43:25.478 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspaces/4fd6069e-1234-abcd-b6b0-abcd1234abcd/>
14:43:26.108 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspaces/4fd6069e-1234-abcd-b6b0-abcd1234abcd/>
Successfully built image '00000000000.dkr.ecr.ap-southeast-1.amazonaws.com/my-repo:latest'
Successfully pushed image '00000000000.dkr.ecr.ap-southeast-1.amazonaws.com/my-repo:latest'
Creating/updating deployments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--14:43:38.429 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspaces/4fd6069e-1234-abcd-b6b0-abcd1234abcd/>
Successfully created/updated all deployments!
Deployments
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Status ┃ Details ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ my-flow/test-ecs-deployment │ applied │ │
└─────────────────────────────┴─────────┴─────────┘
To schedule a run for this deployment, use the following command:
$ prefect deployment run 'my-flow/test-ecs-deployment'
You can also run your flow via the Prefect UI: <https://app.prefect.cloud/account/4206f3f4-c1e7-4076-xxxx-abcd1234abcd/workspace/4fd6069e-1234-abcd-b6b0-abcd1234abcd/deployments/deployment/a358659e-9529-420f-826d-38d45c3bf4f2>
14:43:39.533 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - <AsyncCancelScope, name='coroutine_wrapper' COMPLETED, runtime=14.07> exited
14:43:39.533 | DEBUG | RunSyncEventLoopThread | prefect._internal.concurrency - Finished async call coroutine_wrapper(<dropped>)
I then run it from the cli (can also do it from the UI)
$ prefect deployment run 'my-flow/test-ecs-deployment'
Now, I see from ECS a new task runs, the container starts, then it fails with this error message:
exec /usr/bin/tini: exec format error
I've been struggling with this for quite some time - Marvin has helped a little but, but... still struggling.
(Also, another issue, I see quite frequently it runs the container in mu private subnets - the subnets I ecplicitely set in SUBNETS
are my public ones since I need internet access and I don't want to have to mess with NAT Gateways right now.)
I largely followed the https://docs.prefect.io/integrations/prefect-aws/ecs_guide#1-write-a-simple-test-flow documentation but the last bit about actual deplpyments doesn't go into a whole lot of detail..
And no, I'm not using a prefect.toml - I opted to have all flow deployment specific config in the xxx_deploy.py files. Is this an ok decision to make?
Thanks so much!
(I'm a mac "M" user btw, if that helps)Emilio
02/10/2025, 4:49 PMChris Drumgoole
02/11/2025, 12:15 AMplatform: linux/amd64
in prefect_docker.deployments.steps.build_docker_image:
. Now running into some other issues. making step by step progress!