https://prefect.io logo
Channels
Powered by
r

Ryan Sattler

08/10/2021, 1:32 AM
hi - in my test environment I need to access my S3 Storage (mocked out with Localstack) via a different base URL depending on if I’m registering the container (via my local command line) or actually running it (in local Kubernetes). I’ve tried to configure things like this:
Copy code
home = os.environ.get("HOME")
if home != "/Users/ryan.sattler":
    s3_url = "<http://localstack:4566>"
else:
    s3_url = "<http://localhost:31566>"

flow.storage = S3(bucket="prefect-flows", key="hello-task.py", client_options={
    "endpoint_url": s3_url,
    "aws_access_key_id": "",
    "aws_secret_access_key": ""
})
However this doesn’t seem to work (the localhost url is always being used, so it works for registration but not at runtime), possibly because the value of flow.storage is getting baked-in at registration time? How can I make this dynamic? Unfortunately given my company’s security setup there is no convenient way to use a real S3 bucket (or real container registry or etc) when testing locally.
c

Chris White

08/10/2021, 1:41 AM
Hi Ryan - by default the Flow object is converted to static bytes and deserialized when run, which is why you're seeing a static `s3_url`; to resolve this, you'll want to provide two kwargs to your S3 storage class: • 
stored_as_script=True
to specify that you'd like to store your flow's definition script • 
local_script_path
- the full path to your Flow file that defines your flow; this file will be uploaded to your "S3" bucket With this configuration, the file will be re-parsed at runtime which should pick up your dynamic logic
r

Ryan Sattler

08/10/2021, 1:46 AM
have made those changes but still getting the same error: 
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "<http://localhost:31566/prefect-flows/hello-task.py>"
the code now looks like this:
Copy code
home = os.environ.get("HOME")
if home != "/Users/ryan.sattler":
    s3_url = "<http://localstack:4566>"
else:
    s3_url = "<http://localhost:31566>"

flow.storage = S3(bucket="prefect-flows",
                  key="hello-task.py",
                  stored_as_script=True,
                  local_script_path="test.py",
                  client_options={
                      "endpoint_url": s3_url,
                      "aws_access_key_id": "",
                      "aws_secret_access_key": ""
                  })
I guess the problem is that before the script can even be executed it has to get the code from somewhere which will be the localhost url created at registration time
I can try to work around this by running the registration inside k8s as well, though that’s a bit of a pain
c

Chris White

08/10/2021, 1:56 AM
Oh I see, my apologies; yea the s3 URL is stored in the backend and is referenced by your agent to download the flow so this is trickier than I suggested
r

Ryan Sattler

08/10/2021, 1:57 AM
No worries. I got it working by hackily registering the flow from another k8s pod I had lying around. That’ll be good enough for now (just testing things out) - thanks
c

Chris White

08/10/2021, 3:23 AM
For sure -- this is definitely an interesting new edge case for me; we have some large changes in the works that I believe will alleviate this sort of chicken / egg problem quite nicely, but it's going to be a while before it's all rolled out so stay tuned 😄
2 Views
Powered by