Jezreel Zamora
02/01/2023, 6:12 AMDocker(files={"additional_file.py": "additional_file.py})
Question:
• How we can set additional files that our flows needs using S3 storage with ECSRun configuration?
• If its not possible to set the additional files, is there any recommended workaround ?Jean Luciano
02/01/2023, 4:08 PMJezreel Zamora
02/07/2023, 4:19 AMtest_flow.py
test_data_config.py
Configuration
S3(bucket=bucket)
ECSRun(task_definition_arn=xxx, labels='ecs-agent', cpu=1024, memory=2048)
we are importing test_data_config.py in test_flow.py but if we try to run it, it will cause an import error because that file does not exist in the S3.
If we use Docker storage, we can make this work because we can specify the additional files in files parameter. Is there a way to do this in S3 Storage?Christopher Boyd
02/07/2023, 1:48 PMJezreel Zamora
02/08/2023, 2:42 AMimport test_data_config
from prefect import Flow
with flow(name="test-flow"):
#some code here
When using S3 storage, it only upload the test_flow.py in s3 which why when its running in ECS we are getting an import error.
is there a way to include the test_data_config.py in this scenario?Jean Luciano
02/08/2023, 7:46 PMtest_data_config.py
on top. This because in v1, we pickle the flow file which doesn't support outside dependencies.