Ryan Sattler

    Ryan Sattler

    1 year ago
    hi - in my test environment I need to access my S3 Storage (mocked out with Localstack) via a different base URL depending on if I’m registering the container (via my local command line) or actually running it (in local Kubernetes). I’ve tried to configure things like this:
    home = os.environ.get("HOME")
    if home != "/Users/ryan.sattler":
        s3_url = "<http://localstack:4566>"
    else:
        s3_url = "<http://localhost:31566>"
    
    flow.storage = S3(bucket="prefect-flows", key="hello-task.py", client_options={
        "endpoint_url": s3_url,
        "aws_access_key_id": "",
        "aws_secret_access_key": ""
    })
    However this doesn’t seem to work (the localhost url is always being used, so it works for registration but not at runtime), possibly because the value of flow.storage is getting baked-in at registration time? How can I make this dynamic? Unfortunately given my company’s security setup there is no convenient way to use a real S3 bucket (or real container registry or etc) when testing locally.
    Chris White

    Chris White

    1 year ago
    Hi Ryan - by default the Flow object is converted to static bytes and deserialized when run, which is why you're seeing a static s3_url; to resolve this, you'll want to provide two kwargs to your S3 storage class:
    stored_as_script=True
    to specify that you'd like to store your flow's definition script •
    local_script_path
    - the full path to your Flow file that defines your flow; this file will be uploaded to your "S3" bucket With this configuration, the file will be re-parsed at runtime which should pick up your dynamic logic
    Ryan Sattler

    Ryan Sattler

    1 year ago
    have made those changes but still getting the same error:
    botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "<http://localhost:31566/prefect-flows/hello-task.py>"
    the code now looks like this:
    home = os.environ.get("HOME")
    if home != "/Users/ryan.sattler":
        s3_url = "<http://localstack:4566>"
    else:
        s3_url = "<http://localhost:31566>"
    
    flow.storage = S3(bucket="prefect-flows",
                      key="hello-task.py",
                      stored_as_script=True,
                      local_script_path="test.py",
                      client_options={
                          "endpoint_url": s3_url,
                          "aws_access_key_id": "",
                          "aws_secret_access_key": ""
                      })
    I guess the problem is that before the script can even be executed it has to get the code from somewhere which will be the localhost url created at registration time
    I can try to work around this by running the registration inside k8s as well, though that’s a bit of a pain
    Chris White

    Chris White

    1 year ago
    Oh I see, my apologies; yea the s3 URL is stored in the backend and is referenced by your agent to download the flow so this is trickier than I suggested
    Ryan Sattler

    Ryan Sattler

    1 year ago
    No worries. I got it working by hackily registering the flow from another k8s pod I had lying around. That’ll be good enough for now (just testing things out) - thanks
    Chris White

    Chris White

    1 year ago
    For sure -- this is definitely an interesting new edge case for me; we have some large changes in the works that I believe will alleviate this sort of chicken / egg problem quite nicely, but it's going to be a while before it's all rolled out so stay tuned 😄