Hello Everyone, I have some question in using S3 storage with ECSRun configuration with Prefect V1. Currently we are using Docker storage with DockerRun and we can set additional files that our flow needs by

Docker(files={"additional_file.py": "additional_file.py})

Question: • How we can set additional files that our flows needs using S3 storage with ECSRun configuration? • If its not possible to set the additional files, is there any recommended workaround ?

Have you tried running it? I don't think you'd have to specify the extra files for s3 storage.

Hi @Jean Luciano, I tried running ECSRun with S3 Storage but I’m getting an import error. Our file structure is

test_flow.py
test_data_config.py

Configuration

S3(bucket=bucket)
ECSRun(task_definition_arn=xxx, labels='ecs-agent', cpu=1024, memory=2048)

we are importing test_data_config.py in test_flow.py but if we try to run it, it will cause an import error because that file does not exist in the S3. If we use Docker storage, we can make this work because we can specify the additional files in files parameter. Is there a way to do this in S3 Storage?

You’re running a flow that imports another file, but that file isn’t in storage ?

Hello @Christopher Boyd, Basically, we are importing test_data_config.py from test_flow.py

import test_data_config 
from prefect import Flow

with flow(name="test-flow"):
  #some code here

When using S3 storage, it only upload the test_flow.py in s3 which why when its running in ECS we are getting an import error. is there a way to include the test_data_config.py in this scenario?

Ah I see. The only work around for this is to copy and paste that

test_data_config.py

on top. This because in v1, we pickle the flow file which doesn't support outside dependencies.