sidravic
04/16/2022, 8:59 AMtask_definition_arn
with the containers named as flow
While I'm able to trigger the flows, the flow crashes with the error
copilot/flow/8d31faa7f1ba File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
copilot/flow/8d31faa7f1ba return _bootstrap._gcd_import(name[level:], package, level)
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 991, in _find_and_load
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 991, in _find_and_load
copilot/flow/8d31faa7f1ba File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
copilot/flow/8d31faa7f1ba ModuleNotFoundError: No module named '/root/'
However, i've ensured the flows folder from my project are under the PYTHONPATH and I can't entirely figure out what (if anything) that cloudpickle is trying to do to access those flows at the time of execution.Anna Geller
task_definition_arn
, though - it's mainly focused on task_role_arn
and execution_role_arn
. - can you share your ECSRun?I can't entirely figure out what (if anything) that cloudpickle is trying to do to access those flows at the time of execution.You don't necessarily need to use cloudpickle - check this doc and this example using script storage rather than pickle
sidravic
04/16/2022, 11:10 AMFROM prefecthq/prefect:latest-python3.8
ENV ANALYTICS_APP_DIR /app
WORKDIR ${ANALYTICS_APP_DIR}
RUN apt-get update -y \
&& apt-get update && apt-get install -y curl build-essential libssl-dev libffi-dev lib32ncurses5-dev git libsnappy-dev postgresql-client\
&& apt-get -y install telnet vim unzip
ADD poetry.lock $ANALYTICS_APP_DIR/
ADD pyproject.toml $ANALYTICS_APP_DIR
ENV PYTHONPATH=$ANALYTICS_APP_DIR
ENV POETRY_VIRTUALENVS_PATH=$ANALYTICS_APP_DIR/
RUN pip install poetry
RUN poetry install
ADD . $ANALYTICS_APP_DIR/
CMD ["/bin/bash", "-c", "./ops/launch-ecs-agent.sh"]
ecs_run_config = ECSRun(
labels=[f"{cfg('ENV')}"],
task_definition_arn=task_definition_arn,
task_role_arn=task_role_arn,
execution_role_arn=execution_role_arn,
cpu=1024,
memory=2048,
run_task_kwargs=dict(
cluster=f"{cfg('ECS_CLUSTER_NAME')}",
launchType=f"{cfg('LAUNCH_TYPE')}",
overrides=dict(
containerOverrides=[
dict(name="flow",
command=["poetry", "run", "prefect", "run", "hello-flow"],
cpu=1024,
memory=2048,
)
]
),
),
)