Ben Muller
11/02/2022, 2:20 AMprefect-aws
deps installed.
I have an infrastructure ECSTask
block that pulls a custom image:
FROM prefecthq/prefect:2.6.5-python3.9
RUN pip install prefect-aws s3fs==2022.5.0
This installs everything I need to run a demo flow that is:
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE
from prefect_aws.ecs import ECSTask
ecs_task_block = ECSTask.load("staging-test")
@task
def log_task():
logger = get_run_logger()
<http://logger.info|logger.info>("Hello %s!", "BEN")
<http://logger.info|logger.info>("Prefect Version = %s ๐", prefect.__version__)
logger.debug(AN_IMPORTED_MESSAGE)
@flow
def my_test_flow(name: str):
log_task(name)
if __name__ == "__main__":
my_test_flow()
This flow is using S3 Block Storage, now when I trigger this flow from the cloud UI the task is generated in my cluster, it just fails with a ModuleNotFoundError: No module named 'prefect_aws.ecs'
More details in ๐งต$ docker build . -t prefect
$ docker run -it --name my_prefect prefect /bin/sh
$ pip show prefect-aws
( it is there )
$ python -c "from prefect_aws.ecs import ECSTask"
โ
What am I doing here or not understanding about how storage runs and executes?prefect-aws
version was 0.1.2
So I had to specify 0.1.7. This then caused issues with the s3fs
version I had specified, so I then had to relax those to include any version and it worked.