https://prefect.io logo
Title
d

Danilo Drobac

06/10/2022, 11:39 AM
Has anybody come across an error when using
prefect cloud login --key {key} --workspace {workspace}
related to timeouts? I'm trying to run the command in a CI/CD pipeline through Google Cloud Build but it fails due to a timeout and I have very little information in the error.
a

Anna Geller

06/10/2022, 11:41 AM
There is one caveat: you need to specify your account, not only the workspace name, since the workspace name is not unique without the account name
e.g. my account is annaprefect and workspace is prod, so I would need to use:
prefect cloud login -k API_KEY -w "annaprefect/prod"
d

Danilo Drobac

06/10/2022, 11:42 AM
Yep, I'm doing that.
a

Anna Geller

06/10/2022, 11:42 AM
Can you recreate the API key and ensure it won't expire too soon? definitely worth replacing the API key
d

Danilo Drobac

06/10/2022, 11:43 AM
I've tried the command locally and it works perfectly (just tried it now), but the CI/CD still times out.
a

Anna Geller

06/10/2022, 11:44 AM
looks like your CI is using older Prefect 2.0 version which has older (now incompatible) API version
try using 2.0b6 from your CI
d

Danilo Drobac

06/10/2022, 11:47 AM
This is what I'm getting from my CI builder that I created in a Dockerfile:
Step #2 - "Prefect Version": Version:             2.0b6
Step #2 - "Prefect Version": API version:         0.5.0
Step #2 - "Prefect Version": Python version:      3.9.13
Step #2 - "Prefect Version": Git commit:          502ee008
Step #2 - "Prefect Version": Built:               Tue, Jun 7, 2022 4:58 PM
Step #2 - "Prefect Version": OS/Arch:             linux/x86_64
Step #2 - "Prefect Version": Profile:             default
Step #2 - "Prefect Version": Server type:         ephemeral
Step #2 - "Prefect Version": Server:
Step #2 - "Prefect Version":   Database:          sqlite
Step #2 - "Prefect Version":   SQLite version:    3.34.1
a

Anna Geller

06/10/2022, 11:48 AM
is this a GitHub action?
d

Danilo Drobac

06/10/2022, 11:48 AM
No, it's using Google Cloud Build
a

Anna Geller

06/10/2022, 11:49 AM
what's the error?
d

Danilo Drobac

06/10/2022, 11:50 AM
Starting Step #2 - "Login to Prefect Cloud 2.0"
Step #2 - "Login to Prefect Cloud 2.0": Already have image (with digest): <http://gcr.io/nzyte-modern-data-stack/prefect-builder|gcr.io/nzyte-modern-data-stack/prefect-builder>
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     addr_obj = ip_address(remote_host)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/ipaddress.py", line 53, in ip_address
Step #2 - "Login to Prefect Cloud 2.0":     raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
Step #2 - "Login to Prefect Cloud 2.0": ValueError: '<http://api-beta.prefect.io|api-beta.prefect.io>' does not appear to be an IPv4 or IPv6 address
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     gai_res = await getaddrinfo(
Step #2 - "Login to Prefect Cloud 2.0": asyncio.exceptions.CancelledError
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     yield
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     raise TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     yield
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     raise exc
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     response = await connection.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     raise exc
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     stream = await self._connect(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
Step #2 - "Login to Prefect Cloud 2.0":     stream = await self._network_backend.connect_tcp(**kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     return await self._backend.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     raise to_exc(exc)
Step #2 - "Login to Prefect Cloud 2.0": httpcore.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": The above exception was the direct cause of the following exception:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return fn(*args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 122, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return run_async_in_new_loop(async_fn, *args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 69, in run_async_in_new_loop
Step #2 - "Login to Prefect Cloud 2.0":     return anyio.run(partial(__fn, *args, **kwargs))
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
Step #2 - "Login to Prefect Cloud 2.0":     return asynclib.run(func, *args, **backend_options)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
Step #2 - "Login to Prefect Cloud 2.0":     return native_run(wrapper(), debug=debug)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
Step #2 - "Login to Prefect Cloud 2.0":     return loop.run_until_complete(main)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Step #2 - "Login to Prefect Cloud 2.0":     return future.result()
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return await func(*args)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 204, in login
Step #2 - "Login to Prefect Cloud 2.0":     workspaces = await client.read_workspaces()
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 82, in read_workspaces
Step #2 - "Login to Prefect Cloud 2.0":     return await self.get("/me/workspaces")
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 102, in get
Step #2 - "Login to Prefect Cloud 2.0":     res = await self._client.get(route, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
Step #2 - "Login to Prefect Cloud 2.0":     return await self.request(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
Step #2 - "Login to Prefect Cloud 2.0":     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_handling_auth(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_handling_redirects(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_single_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
Step #2 - "Login to Prefect Cloud 2.0":     response = await transport.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     raise mapped_exc(message) from exc
Step #2 - "Login to Prefect Cloud 2.0": httpx.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": An exception occurred.
a

Anna Geller

06/10/2022, 12:14 PM
can you share Dockerfile you used for gcr.io/nzyte-modern-data-stack/prefect-builder?
in general, you don't need to run this in Docker, we run all tests on Ubuntu, so this might be easier to set up
you would only need to install this in your CI:
pip install gcsfs
pip install -U "prefect>=2.0b"
all other dependencies from your Docker image can be used at runtime in your DockerFlowRunner or KubernetesFlowRunner, but they don't need to be available during deployment creation process
I think since you mentioned the same cloud login command login works locally, my feeling is there is some weird network issue given that you run this in a Docker container, then Google Cloud Build may itself run in a Docker container and it's somehow causes issues it's worth trying to start CI with the latest Ubuntu version and install Prefect as part of CI before creating deployment
d

Danilo Drobac

06/10/2022, 12:39 PM
The Dockerfile is based on python:3.9 and just installs those 2 dependencies. I was struggling to get the prefect installation to persist through different steps in the build pipeline, so I first tried to do it all in a single step that started from Python3.9, installed the dependencies and then logged into the workspace but that gave the same timeout error. I agree with you that it may be something network related but the lack of error message makes it difficult. I'm going to try and ping the host (api-beta.prefect.io) to see if it succeeds, that should be a good first step.
a

Anna Geller

06/10/2022, 1:29 PM
Yes, for sure. What do you mean by multiple steps in the pipeline? all you need is to login to cloud and create a deployment
curious to know what your CI consists of
btw you can also use Prefect 2.0 base image instead of Python 3.9 - just something you may consider:
FROM prefecthq/prefect:2.0b6-python3.9
d

Danilo Drobac

06/10/2022, 2:25 PM
When I say steps in the pipeline I mean the entire CI Build pipeline. Step 1: get list of flows from the flows folder and store in a text file. Step 2: install dependencies for prefect Step 3: login to prefect cloud Step 4: deploy each of the flows
Similar to what you have in the GitHub actions script you shared, although I don't need to authenticate with GCP before the deployment when I'm in Cloud Build
But yeah, the challenge I'm facing is that I do the steps as listed above, step 3 complains that prefect isn't installed in the Ubuntu or python images. I can't use a docker image directly in Build, I have to use an existing "builder" or submit my own build via Dockerfile to Google Container Repository and then use that image, which is what I've tried but get the timeout error.
a

Anna Geller

06/10/2022, 2:31 PM
Can you share your approach step by step e.g. in a Markdown Gist with screenshots showing what you did? Perhaps this way I could maybe even reproduce, even though I've never used cloud build, or at least see a bigger picture Also you can try using Prefect base image as shown above
d

Danilo Drobac

06/10/2022, 3:35 PM
Okay, I just tried to change to the Prefect base image instead and it still produces the same error message. The Dockerfile is now as simple as:
FROM prefecthq/prefect:2.0b6-python3.9
And it works correctly, because
prefect version
is returning the correct values.
I'll try and put everything together into a Gist for you to see if you can reproduce it yourself.
But currently, the extent of the
cloudbuild.yaml
file is:
steps:
- id: 'Generate full list of flows/DeploymentSpecs'
  name: 'ubuntu'
  entrypoint: 'printf'
  args:
    - "'%s\n' workspace/flows/* > flow_list.txt"

- id: 'Login to Prefect Cloud 2.0'
  name: '<http://gcr.io/{project-name}/{image_name}|gcr.io/{project-name}/{image_name}>'
  args: ['prefect', 'cloud', 'login', '--key', '{insert_key}', '--workspace', '{insert_workspace}']
Another update, if I add the
prefect cloud login
command to the Dockerfile, it successfully logs in. I'm going to see whether the deploy command works as expected when the authorisation is handled in the image. It feels like it's not the correct way of doing it, but if it works then at least I've found a workaround.
Interestingly when I do this, I get the following which says that it is using FileStorage as expected, but I don't see it within the Prefect Cloud 2.0 Workspace...
Step #2 - "Create deployment": Loading deployment specifications from python script at 
Step #2 - "Create deployment": 'flows/second_flow.py'...
Step #2 - "Create deployment": Creating deployment 'second-deployment' for flow 'main-flow'...
Step #2 - "Create deployment": Deploying flow script from '/workspace/flows/second_flow.py' using File 
Step #2 - "Create deployment": Storage...
Step #2 - "Create deployment": Created deployment 'main-flow/second-deployment'.
Step #2 - "Create deployment": View your new deployment with: 
Step #2 - "Create deployment": 
Step #2 - "Create deployment":     prefect deployment inspect 'main-flow/second-deployment'
Step #2 - "Create deployment": Created 1 deployments!
Finished Step #2 - "Create deployment"
2022/06/10 17:58:10 Step Step #2 - "Create deployment" finished
2022/06/10 17:58:11 status changed to "DONE"
a

Anna Geller

06/10/2022, 7:55 PM
This Gist is amazing, thank you so much! What I meant was that you try to do this instead:
steps:
- id: 'Generate full list of flows/DeploymentSpecs'
  name: 'ubuntu'
  entrypoint: 'printf'
  args:
    - "'%s\n' workspace/flows/* > flow_list.txt"

- id: 'Login to Prefect Cloud 2.0'
  name: 'prefecthq/prefect:2.0b6-python3.9'
  args: ['prefect', 'cloud', 'login', '--key', '{insert_key}', '--workspace', '{insert_workspace}']
and great that you at least have a workaround
d

Danilo Drobac

06/13/2022, 10:23 AM
Tried doing as you suggested by using the image directly and again I get the exact same timeout error. Very puzzling.
And unfortunately, I haven't quite found a workaround because even though I can login to the workspace with the method I've used, when I run the deployment (which succeeds, and says it's using FileStorage), the deployment doesn't show up in Prefect Cloud... 🤔
a

Anna Geller

06/13/2022, 11:09 AM
I honestly don't know why you can't login from Cloud Build CI - can you try a similar pipeline from GitHub actions as I originally showed? This way, you can confirm whether this is an issue with the code Build setup - I can't replicate your issue on GHA so it may be something with the code build or docker setup
d

Danilo Drobac

06/13/2022, 11:13 AM
I'm going to post in the Google Slack to see if I can get any further from that avenue. Will report back. Thanks for all your help Anna!
🙌 1
a

Anna Geller

06/13/2022, 11:15 AM
Great idea!
d

Danilo Drobac

06/16/2022, 11:38 AM
Got it working perfectly with GitHub actions 😎 There's also another mechanism you can use for authentication with GCP that doesn't involve uploading a service account file (better practice). You can read about it here: https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions And the final actions script I have looks like this:
name: CI
on:
  push:
    branches: [ main ]
jobs:
  list-flows:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v3
      - id: set-matrix
        run: echo "::set-output name=matrix::$(ls flows/*.py | jq -R -s -c 'split("\n")[:-1]')"
  deploy:
    needs: list-flows
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    strategy:
      matrix:
        flows: ${{ fromJson(needs.list-flows.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.9
        uses: actions/setup-python@v3
        with:
          python-version: 3.9
      - name: Dev dependencies
        run: |
          pip install gcsfs
          pip install -U "prefect>=2.0b"
      - name: Login to Prefect Cloud 2.0
        run: prefect cloud login --key ${{ secrets.PREFECT_API_KEY }} --workspace 'danilodrobacnzytecouk/nzyte-prefect'
      - name: Validate Prefect version
        run: prefect version
      - name: Authenticate to GCP
        uses: 'google-github-actions/auth@v0.7.1'
        with:
          workload_identity_provider: 'projects/170961541935/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
          service_account: '<mailto:prefect-service-account@nzyte-modern-data-stack.iam.gserviceaccount.com|prefect-service-account@nzyte-modern-data-stack.iam.gserviceaccount.com>'
      - name: Create deployment
        run: prefect deployment create ${{ matrix.flows }}
💯 1
🙌 1
a

Anna Geller

06/16/2022, 11:43 AM
This is awesome, thanks so much for sharing! I read about it, but then I considered many users already have a JSON file with SA permissions so I assume this is an easier way, but yeah definitely IAM roles ftw 🙌
🙊 1
d

Danilo Drobac

06/16/2022, 11:58 AM
Ahh not so great news when using FileStorage.
The bq and gsutil tools do not currently support Workload Identity Federation! You will need to use traditional service account key authentication for now.
So looks like you still need to use a service account for authentication when interfacing with GCS or BQ. Typical, but still one to look out for when they update it!
a

Anna Geller

06/16/2022, 12:04 PM
wow, this is surprising, thanks for giving this update!
I honestly don't know why GCS makes permissioning so hard, it's so much easier on AWS
d

Danilo Drobac

06/16/2022, 12:06 PM
I've never used AWS but it's definitely a more mature product. Workload Identity is a new (as far as I can tell) feature in GCP and
bq
and
gsutil
are legacy packages, I guess they just haven't gotten round to updating the codebase, although you would think that should be part of the rollout of the new product itself so people don't run in to issues like this!
👍 1
a

Anna Geller

06/16/2022, 1:19 PM
wow, I'm learning a lot from you here. I thought workload identity is basically IAM roles under the hood but seems I was wrong thanks!