Jacob Wilson
03/15/2022, 4:31 PMSubmitted for execution: phase. I am using Docker storage (The image is hosted in ECR and I’ve confirmed my execution role has access to the repo) and ECS Run. The flow runs locally but not in ECS.
Dockerfile:
FROM prefecthq/prefect:latest-python3.8
WORKDIR /opt/prefect
COPY requirements.txt .
RUN pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
Flow code:
import os
from prefect import Flow, task
from prefect.run_configs import ECSRun
from prefect.storage import Docker
@task(log_stdout=True)
def extract():
x = [4, 5, 6]
print("Starting: {}".format(x))
return x
@task
def transform(y):
return [i * 10 for i in y]
@task(log_stdout=True)
def load(z):
print("Received: {}".format(z))
with Flow("Test Flow", storage=Docker(dockerfile="Dockerfile", registry_url=os.getenv("REGISTRY_URL"), image_name=os.getenv("IMAGE_NAME"))) as flow:
e = extract()
t = transform(e)
l = load(t)
flow.run_config = ECSRun(
task_role_arn=os.getenv("TASK_ROLE_ARN"),
execution_role_arn=os.getenv("EXECUTION_ROLE_ARN")
)Kevin Kho
Kevin Kho