Hi Community me again I have a strange one I have set up an Prefect Community #prefect-community

Hi Community - me again. I have a strange one, I ...

Ben Muller

11/02/2022, 2:20 AM

Hi Community - me again. I have a strange one, I have set up an agent in ECS with a custom image that has

prefect-aws

deps installed. I have an infrastructure

ECSTask

block that pulls a custom image:

Copy code

FROM        prefecthq/prefect:2.6.5-python3.9

RUN         pip install prefect-aws s3fs==2022.5.0

This installs everything I need to run a demo flow that is:

Copy code

import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE

from prefect_aws.ecs import ECSTask

ecs_task_block = ECSTask.load("staging-test")


@task
def log_task():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello %s!", "BEN")
    <http://logger.info|logger.info>("Prefect Version = %s 🚀", prefect.__version__)
    logger.debug(AN_IMPORTED_MESSAGE)


@flow
def my_test_flow(name: str):
    log_task(name)


if __name__ == "__main__":
    my_test_flow()

This flow is using S3 Block Storage, now when I trigger this flow from the cloud UI the task is generated in my cluster, it just fails with a

ModuleNotFoundError: No module named 'prefect_aws.ecs'

More details in 🧵

✅ 1

Ben Muller

11/02/2022, 2:22 AM

Steps to reproduce locally with the Dockerfile upvote

$ docker build  . -t prefect

$ docker run -it --name my_prefect prefect /bin/sh

$ pip show prefect-aws

( it is there )

$ python -c "from prefect_aws.ecs import ECSTask"

❌ What am I doing here or not understanding about how storage runs and executes?

Ben Muller

11/02/2022, 2:42 AM

Ok - I think I have this solved. For anyone else out there that runs into a similar issue, the default

prefect-aws

version was 0.1.2 So I had to specify 0.1.7. This then caused issues with the

s3fs

version I had specified, so I then had to relax those to include any version and it worked.

🙌 2

🙏 1

Open in Slack

Previous Next