Hi, Could you please help with configuring DockerF...
# prefect-community
Hi, Could you please help with configuring DockerFlowRunner to access flows from GCP storage? I’m following Perfect 2.0 tutorial. I configured GCP storage and successfully used it with “universal runner” for non-dockerised flows. But when I try to run flow deployed with a DockerRunner it fails with the following error cause it couldn’t access the GCP(exception is in the thread). I’m using default user authentification for GCP. Setting
env var to point to a default location (
) before running an agent didn’t help.
Copy code
14:24:47.426 | ERROR   | Flow run 'proficient-wren' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 200, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_deployment(deployment, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 82, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 575, in load_flow_from_deployment
    maybe_flow = await client.resolve_datadoc(deployment.flow_data)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1680, in resolve_datadoc
    return await resolve_inner(datadoc)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1673, in resolve_inner
    data = await self.retrieve_data(data)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1238, in retrieve_data
    storage_block = await self.read_block(block_id)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 909, in read_block
    return create_block_from_api_block(response.json())
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 127, in create_block_from_api_block
    block = block_spec_cls(**api_block.data)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 72, in __init__
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/storage.py", line 270, in block_initialization
    self.storage_client = gcs.Client(project=self.project)
  File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/client.py", line 161, in __init__
    super(Client, self).__init__(
  File "/usr/local/lib/python3.10/site-packages/google/cloud/client/__init__.py", line 321, in __init__
  File "/usr/local/lib/python3.10/site-packages/google/cloud/client/__init__.py", line 178, in __init__
    credentials, _ = google.auth.default(scopes=scopes)
  File "/usr/local/lib/python3.10/site-packages/google/auth/_default.py", line 577, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started>
Is that credentials pointing inside the local container or to the host machine?
It’s inside host machine.
You need the credentials in the container because it will look for this relative to the container
Or you could use the volumes argument to pass a volume mount to the Docker FlowRunner so that it can use the credentials from the local host
Thanks, I’ll try to mount via volume. Where can I figure out what should be the destination path inside the container? I imagine default user is different inside it.
You specify one and then point the ENV VAR to it as long as it matches
Does it mean that I need to build my image anyway? As far as I understand DockerRunner uses
image by default an I’m not sure how to figure out which username is used in this image
No because the mount can be on any image. You can do exec into the container and then poke around navigating the directories
Thanks, I’ll try
Managed to make it run. It is just a
Copy code
If I’m not the only one who faced similar issue, maybe it’s worth adding a note in the tutorial on how to setup credentials for the default Docker container.
Would you like to try making a PR for that?
I would like to try 🙂 It will be my first PR to OSS project so it could take time to figure out the process. Is there somewhere a guide on how to contribute?
We have for Prefect 1 here. For 2.0, which you are using, it’s here