https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
t

Taras Svirskyi

05/13/2022, 4:05 PM
Hi, Could you please help with configuring DockerFlowRunner to access flows from GCP storage? I’m following Perfect 2.0 tutorial. I configured GCP storage and successfully used it with “universal runner” for non-dockerised flows. But when I try to run flow deployed with a DockerRunner it fails with the following error cause it couldn’t access the GCP(exception is in the thread). I’m using default user authentification for GCP. Setting 
GOOGLE_APPLICATION_CREDENTIALS
env var to point to a default location (
/Users/test/.config/gcloud/application_default_credentials.json
) before running an agent didn’t help.
✅ 1
Copy code
14:24:47.426 | ERROR   | Flow run 'proficient-wren' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 200, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_deployment(deployment, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 82, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 575, in load_flow_from_deployment
    maybe_flow = await client.resolve_datadoc(deployment.flow_data)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1680, in resolve_datadoc
    return await resolve_inner(datadoc)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1673, in resolve_inner
    data = await self.retrieve_data(data)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1238, in retrieve_data
    storage_block = await self.read_block(block_id)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 909, in read_block
    return create_block_from_api_block(response.json())
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 127, in create_block_from_api_block
    block = block_spec_cls(**api_block.data)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 72, in __init__
    self.block_initialization()
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/storage.py", line 270, in block_initialization
    self.storage_client = gcs.Client(project=self.project)
  File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/client.py", line 161, in __init__
    super(Client, self).__init__(
  File "/usr/local/lib/python3.10/site-packages/google/cloud/client/__init__.py", line 321, in __init__
    Client.__init__(
  File "/usr/local/lib/python3.10/site-packages/google/cloud/client/__init__.py", line 178, in __init__
    credentials, _ = google.auth.default(scopes=scopes)
  File "/usr/local/lib/python3.10/site-packages/google/auth/_default.py", line 577, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started>
k

Kevin Kho

05/13/2022, 4:12 PM
Is that credentials pointing inside the local container or to the host machine?
t

Taras Svirskyi

05/13/2022, 4:13 PM
It’s inside host machine.
k

Kevin Kho

05/13/2022, 4:15 PM
You need the credentials in the container because it will look for this relative to the container
Or you could use the volumes argument to pass a volume mount to the Docker FlowRunner so that it can use the credentials from the local host
t

Taras Svirskyi

05/13/2022, 4:17 PM
Thanks, I’ll try to mount via volume. Where can I figure out what should be the destination path inside the container? I imagine default user is different inside it.
k

Kevin Kho

05/13/2022, 4:18 PM
You specify one and then point the ENV VAR to it as long as it matches
t

Taras Svirskyi

05/13/2022, 4:20 PM
Does it mean that I need to build my image anyway? As far as I understand DockerRunner uses 
prefecthq/prefect:2.0b3-python3.10
image by default an I’m not sure how to figure out which username is used in this image
k

Kevin Kho

05/13/2022, 4:28 PM
No because the mount can be on any image. You can do exec into the container and then poke around navigating the directories
t

Taras Svirskyi

05/13/2022, 4:34 PM
Thanks, I’ll try
Managed to make it run. It is just a 
root
user
Copy code
DeploymentSpec(
    name="docker-example",
    flow=my_docker_flow,
    flow_runner=DockerFlowRunner(volumes=['/Users/test/.config/gcloud:/root/.config/gcloud']),
    tags=['tutorial','dev'],
)
If I’m not the only one who faced similar issue, maybe it’s worth adding a note in the tutorial on how to setup credentials for the default Docker container.
k

Kevin Kho

05/13/2022, 5:11 PM
Would you like to try making a PR for that?
t

Taras Svirskyi

05/13/2022, 5:17 PM
I would like to try 🙂 It will be my first PR to OSS project so it could take time to figure out the process. Is there somewhere a guide on how to contribute?
k

Kevin Kho

05/13/2022, 6:16 PM
We have for Prefect 1 here. For 2.0, which you are using, it’s here
3 Views
Powered by