Jacob Wilson
03/15/2022, 4:31 PMSubmitted for execution:
phase. I am using Docker
storage (The image is hosted in ECR and I’ve confirmed my execution role has access to the repo) and ECS Run
. The flow runs locally but not in ECS.
Dockerfile:
FROM prefecthq/prefect:latest-python3.8
WORKDIR /opt/prefect
COPY requirements.txt .
RUN pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
Flow code:
import os
from prefect import Flow, task
from prefect.run_configs import ECSRun
from prefect.storage import Docker
@task(log_stdout=True)
def extract():
x = [4, 5, 6]
print("Starting: {}".format(x))
return x
@task
def transform(y):
return [i * 10 for i in y]
@task(log_stdout=True)
def load(z):
print("Received: {}".format(z))
with Flow("Test Flow", storage=Docker(dockerfile="Dockerfile", registry_url=os.getenv("REGISTRY_URL"), image_name=os.getenv("IMAGE_NAME"))) as flow:
e = extract()
t = transform(e)
l = load(t)
flow.run_config = ECSRun(
task_role_arn=os.getenv("TASK_ROLE_ARN"),
execution_role_arn=os.getenv("EXECUTION_ROLE_ARN")
)
Kevin Kho
03/15/2022, 4:34 PM