Ryan Sattler
08/10/2021, 1:32 AMhome = os.environ.get("HOME")
if home != "/Users/ryan.sattler":
s3_url = "<http://localstack:4566>"
else:
s3_url = "<http://localhost:31566>"
flow.storage = S3(bucket="prefect-flows", key="hello-task.py", client_options={
"endpoint_url": s3_url,
"aws_access_key_id": "",
"aws_secret_access_key": ""
})
However this doesn’t seem to work (the localhost url is always being used, so it works for registration but not at runtime), possibly because the value of flow.storage is getting baked-in at registration time? How can I make this dynamic?
Unfortunately given my company’s security setup there is no convenient way to use a real S3 bucket (or real container registry or etc) when testing locally.Chris White
08/10/2021, 1:41 AMstored_as_script=True
to specify that you'd like to store your flow's definition script
• local_script_path
- the full path to your Flow file that defines your flow; this file will be uploaded to your "S3" bucket
With this configuration, the file will be re-parsed at runtime which should pick up your dynamic logicRyan Sattler
08/10/2021, 1:46 AMbotocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "<http://localhost:31566/prefect-flows/hello-task.py>"
home = os.environ.get("HOME")
if home != "/Users/ryan.sattler":
s3_url = "<http://localstack:4566>"
else:
s3_url = "<http://localhost:31566>"
flow.storage = S3(bucket="prefect-flows",
key="hello-task.py",
stored_as_script=True,
local_script_path="test.py",
client_options={
"endpoint_url": s3_url,
"aws_access_key_id": "",
"aws_secret_access_key": ""
})
Chris White
08/10/2021, 1:56 AMRyan Sattler
08/10/2021, 1:57 AMChris White
08/10/2021, 3:23 AM