https://prefect.io logo
Title
j

Jezreel Zamora

02/01/2023, 6:12 AM
Hello Everyone, I have some question in using S3 storage with ECSRun configuration with Prefect V1. Currently we are using Docker storage with DockerRun and we can set additional files that our flow needs by
Docker(files={"additional_file.py": "additional_file.py})
Question: • How we can set additional files that our flows needs using S3 storage with ECSRun configuration? • If its not possible to set the additional files, is there any recommended workaround ?
j

Jean Luciano

02/01/2023, 4:08 PM
Have you tried running it? I don't think you'd have to specify the extra files for s3 storage.
j

Jezreel Zamora

02/07/2023, 4:19 AM
Hi @Jean Luciano, I tried running ECSRun with S3 Storage but I’m getting an import error. Our file structure is
test_flow.py
test_data_config.py
Configuration
S3(bucket=bucket)
ECSRun(task_definition_arn=xxx, labels='ecs-agent', cpu=1024, memory=2048)
we are importing test_data_config.py in test_flow.py but if we try to run it, it will cause an import error because that file does not exist in the S3. If we use Docker storage, we can make this work because we can specify the additional files in files parameter. Is there a way to do this in S3 Storage?
c

Christopher Boyd

02/07/2023, 1:48 PM
You’re running a flow that imports another file, but that file isn’t in storage ?
What’s your expected behavior here ?
Where should it import from ?
j

Jezreel Zamora

02/08/2023, 2:42 AM
Hello @Christopher Boyd, Basically, we are importing test_data_config.py from test_flow.py
import test_data_config 
from prefect import Flow

with flow(name="test-flow"):
  #some code here
When using S3 storage, it only upload the test_flow.py in s3 which why when its running in ECS we are getting an import error. is there a way to include the test_data_config.py in this scenario?
j

Jean Luciano

02/08/2023, 7:46 PM
Ah I see. The only work around for this is to copy and paste that
test_data_config.py
on top. This because in v1, we pickle the flow file which doesn't support outside dependencies.