Thread
#prefect-community
    Taras Svirskyi

    Taras Svirskyi

    4 months ago
    Hi, Could you please help with configuring DockerFlowRunner to access flows from GCP storage? I’m following Perfect 2.0 tutorial. I configured GCP storage and successfully used it with “universal runner” for non-dockerised flows. But when I try to run flow deployed with a DockerRunner it fails with the following error cause it couldn’t access the GCP(exception is in the thread). I’m using default user authentification for GCP. Setting
    GOOGLE_APPLICATION_CREDENTIALS
    env var to point to a default location (
    /Users/test/.config/gcloud/application_default_credentials.json
    ) before running an agent didn’t help.
    14:24:47.426 | ERROR   | Flow run 'proficient-wren' - Flow could not be retrieved from deployment.
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 200, in retrieve_flow_then_begin_flow_run
        flow = await load_flow_from_deployment(deployment, client=client)
      File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 82, in with_injected_client
        return await fn(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 575, in load_flow_from_deployment
        maybe_flow = await client.resolve_datadoc(deployment.flow_data)
      File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1680, in resolve_datadoc
        return await resolve_inner(datadoc)
      File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1673, in resolve_inner
        data = await self.retrieve_data(data)
      File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1238, in retrieve_data
        storage_block = await self.read_block(block_id)
      File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 909, in read_block
        return create_block_from_api_block(response.json())
      File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 127, in create_block_from_api_block
        block = block_spec_cls(**api_block.data)
      File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 72, in __init__
        self.block_initialization()
      File "/usr/local/lib/python3.10/site-packages/prefect/blocks/storage.py", line 270, in block_initialization
        self.storage_client = gcs.Client(project=self.project)
      File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/client.py", line 161, in __init__
        super(Client, self).__init__(
      File "/usr/local/lib/python3.10/site-packages/google/cloud/client/__init__.py", line 321, in __init__
        Client.__init__(
      File "/usr/local/lib/python3.10/site-packages/google/cloud/client/__init__.py", line 178, in __init__
        credentials, _ = google.auth.default(scopes=scopes)
      File "/usr/local/lib/python3.10/site-packages/google/auth/_default.py", line 577, in default
        raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
    google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started>
    Kevin Kho

    Kevin Kho

    4 months ago
    Is that credentials pointing inside the local container or to the host machine?
    Taras Svirskyi

    Taras Svirskyi

    4 months ago
    It’s inside host machine.
    Kevin Kho

    Kevin Kho

    4 months ago
    You need the credentials in the container because it will look for this relative to the container
    Or you could use the volumes argument to pass a volume mount to the Docker FlowRunner so that it can use the credentials from the local host
    Taras Svirskyi

    Taras Svirskyi

    4 months ago
    Thanks, I’ll try to mount via volume. Where can I figure out what should be the destination path inside the container? I imagine default user is different inside it.
    Kevin Kho

    Kevin Kho

    4 months ago
    You specify one and then point the ENV VAR to it as long as it matches
    Taras Svirskyi

    Taras Svirskyi

    4 months ago
    Does it mean that I need to build my image anyway? As far as I understand DockerRunner uses
    prefecthq/prefect:2.0b3-python3.10
    image by default an I’m not sure how to figure out which username is used in this image
    Kevin Kho

    Kevin Kho

    4 months ago
    No because the mount can be on any image. You can do exec into the container and then poke around navigating the directories
    Taras Svirskyi

    Taras Svirskyi

    4 months ago
    Thanks, I’ll try
    Managed to make it run. It is just a
    root
    user
    DeploymentSpec(
        name="docker-example",
        flow=my_docker_flow,
        flow_runner=DockerFlowRunner(volumes=['/Users/test/.config/gcloud:/root/.config/gcloud']),
        tags=['tutorial','dev'],
    )
    If I’m not the only one who faced similar issue, maybe it’s worth adding a note in the tutorial on how to setup credentials for the default Docker container.
    Kevin Kho

    Kevin Kho

    4 months ago
    Would you like to try making a PR for that?
    Taras Svirskyi

    Taras Svirskyi

    4 months ago
    I would like to try 🙂 It will be my first PR to OSS project so it could take time to figure out the process. Is there somewhere a guide on how to contribute?
    Kevin Kho

    Kevin Kho

    4 months ago
    We have for Prefect 1 here. For 2.0, which you are using, it’s here