Mark McDonald01/21/2021, 11:53 PM
to be an image you’ve setup to contain the additional code you require.
Mark McDonald01/22/2021, 12:04 AM
image contain the helpers then the flow can still be stored in S3 and executed in the helper image.
Mark McDonald01/22/2021, 12:07 AM
that pulls down the requirements -- this is a bit of a hack though.
that runs arbitrary commands before executing the flow run
Mark McDonald01/22/2021, 12:43 AM
Basically, I think the idea of S3 Storage is appealing because if only flow code changes (not dependencies or helpers), then I can skip steps 1 and 2 during my CI/CD. I just have to call S3 storage on the single flow script that's changed, register the flow and you all take care of it from there. Subclassing S3Storage seems like it might work, but I agree that it doesn't feel right. I would imagine that other Prefect S3 storage users would want to have the ability to define the flow's location within their image as well. I think this should be configurable. Docker storage offers the configuration through the
# step 1: docker build/push/tag image # step 2: create ecs task definition # step 3: flow.storage = S3( bucket=S3_BUCKET, key=s3_key, stored_as_script=True, local_script_path=/path/to/flow.py, ) # step 4: flow.run_config = ECSRun( task_definition_arn=task_definition_arn, run_task_kwargs=run_task_kwargs ) # step 5: flow_id = flow.register( labels=['dev'], project_name=PROJECT_NAME, )
argument. Can I propose that this arg be added to s3 storage? https://github.com/PrefectHQ/prefect/blob/c8d9b9b7a6d11b9487901cd795b8f1509f355845/src/prefect/storage/docker.py#L108-L109