https://prefect.io logo
Join Slack
Powered by
Has anybody come across an error when using `prefe...
# ask-community
d
Has anybody come across an error when using 
prefect cloud login --key {key} --workspace {workspace}
related to timeouts? I'm trying to run the command in a CI/CD pipeline through Google Cloud Build but it fails due to a timeout and I have very little information in the error.
a
There is one caveat: you need to specify your account, not only the workspace name, since the workspace name is not unique without the account name
e.g. my account is annaprefect and workspace is prod, so I would need to use:
Copy code
prefect cloud login -k API_KEY -w "annaprefect/prod"
d
Yep, I'm doing that.
a
Can you recreate the API key and ensure it won't expire too soon? definitely worth replacing the API key
d
I've tried the command locally and it works perfectly (just tried it now), but the CI/CD still times out.
a
looks like your CI is using older Prefect 2.0 version which has older (now incompatible) API version
try using 2.0b6 from your CI
d
This is what I'm getting from my CI builder that I created in a Dockerfile:
Copy code
Step #2 - "Prefect Version": Version:             2.0b6
Step #2 - "Prefect Version": API version:         0.5.0
Step #2 - "Prefect Version": Python version:      3.9.13
Step #2 - "Prefect Version": Git commit:          502ee008
Step #2 - "Prefect Version": Built:               Tue, Jun 7, 2022 4:58 PM
Step #2 - "Prefect Version": OS/Arch:             linux/x86_64
Step #2 - "Prefect Version": Profile:             default
Step #2 - "Prefect Version": Server type:         ephemeral
Step #2 - "Prefect Version": Server:
Step #2 - "Prefect Version":   Database:          sqlite
Step #2 - "Prefect Version":   SQLite version:    3.34.1
a
is this a GitHub action?
d
No, it's using Google Cloud Build
a
what's the error?
d
Copy code
Starting Step #2 - "Login to Prefect Cloud 2.0"
Step #2 - "Login to Prefect Cloud 2.0": Already have image (with digest): <http://gcr.io/nzyte-modern-data-stack/prefect-builder|gcr.io/nzyte-modern-data-stack/prefect-builder>
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     addr_obj = ip_address(remote_host)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/ipaddress.py", line 53, in ip_address
Step #2 - "Login to Prefect Cloud 2.0":     raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
Step #2 - "Login to Prefect Cloud 2.0": ValueError: '<http://api-beta.prefect.io|api-beta.prefect.io>' does not appear to be an IPv4 or IPv6 address
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     gai_res = await getaddrinfo(
Step #2 - "Login to Prefect Cloud 2.0": asyncio.exceptions.CancelledError
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     yield
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     raise TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": TimeoutError
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": During handling of the above exception, another exception occurred:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     yield
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     raise exc
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     response = await connection.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     raise exc
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     stream = await self._connect(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
Step #2 - "Login to Prefect Cloud 2.0":     stream = await self._network_backend.connect_tcp(**kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     return await self._backend.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
Step #2 - "Login to Prefect Cloud 2.0":     stream: anyio.abc.ByteStream = await anyio.connect_tcp(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     raise to_exc(exc)
Step #2 - "Login to Prefect Cloud 2.0": httpcore.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": The above exception was the direct cause of the following exception:
Step #2 - "Login to Prefect Cloud 2.0": 
Step #2 - "Login to Prefect Cloud 2.0": Traceback (most recent call last):
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return fn(*args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 122, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return run_async_in_new_loop(async_fn, *args, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncio.py", line 69, in run_async_in_new_loop
Step #2 - "Login to Prefect Cloud 2.0":     return anyio.run(partial(__fn, *args, **kwargs))
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
Step #2 - "Login to Prefect Cloud 2.0":     return asynclib.run(func, *args, **backend_options)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
Step #2 - "Login to Prefect Cloud 2.0":     return native_run(wrapper(), debug=debug)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
Step #2 - "Login to Prefect Cloud 2.0":     return loop.run_until_complete(main)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Step #2 - "Login to Prefect Cloud 2.0":     return future.result()
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
Step #2 - "Login to Prefect Cloud 2.0":     return await func(*args)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 204, in login
Step #2 - "Login to Prefect Cloud 2.0":     workspaces = await client.read_workspaces()
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 82, in read_workspaces
Step #2 - "Login to Prefect Cloud 2.0":     return await self.get("/me/workspaces")
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/prefect/cli/cloud.py", line 102, in get
Step #2 - "Login to Prefect Cloud 2.0":     res = await self._client.get(route, **kwargs)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
Step #2 - "Login to Prefect Cloud 2.0":     return await self.request(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
Step #2 - "Login to Prefect Cloud 2.0":     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_handling_auth(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_handling_redirects(
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
Step #2 - "Login to Prefect Cloud 2.0":     response = await self._send_single_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
Step #2 - "Login to Prefect Cloud 2.0":     response = await transport.handle_async_request(request)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Step #2 - "Login to Prefect Cloud 2.0":     resp = await self._pool.handle_async_request(req)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Step #2 - "Login to Prefect Cloud 2.0":     self.gen.throw(typ, value, traceback)
Step #2 - "Login to Prefect Cloud 2.0":   File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
Step #2 - "Login to Prefect Cloud 2.0":     raise mapped_exc(message) from exc
Step #2 - "Login to Prefect Cloud 2.0": httpx.ConnectTimeout
Step #2 - "Login to Prefect Cloud 2.0": An exception occurred.
a
can you share Dockerfile you used for gcr.io/nzyte-modern-data-stack/prefect-builder?
in general, you don't need to run this in Docker, we run all tests on Ubuntu, so this might be easier to set up
you would only need to install this in your CI:
Copy code
pip install gcsfs
pip install -U "prefect>=2.0b"
all other dependencies from your Docker image can be used at runtime in your DockerFlowRunner or KubernetesFlowRunner, but they don't need to be available during deployment creation process
I think since you mentioned the same cloud login command login works locally, my feeling is there is some weird network issue given that you run this in a Docker container, then Google Cloud Build may itself run in a Docker container and it's somehow causes issues it's worth trying to start CI with the latest Ubuntu version and install Prefect as part of CI before creating deployment
d
The Dockerfile is based on python:3.9 and just installs those 2 dependencies. I was struggling to get the prefect installation to persist through different steps in the build pipeline, so I first tried to do it all in a single step that started from Python3.9, installed the dependencies and then logged into the workspace but that gave the same timeout error. I agree with you that it may be something network related but the lack of error message makes it difficult. I'm going to try and ping the host (api-beta.prefect.io) to see if it succeeds, that should be a good first step.
a
Yes, for sure. What do you mean by multiple steps in the pipeline? all you need is to login to cloud and create a deployment
curious to know what your CI consists of
btw you can also use Prefect 2.0 base image instead of Python 3.9 - just something you may consider:
Copy code
FROM prefecthq/prefect:2.0b6-python3.9
d
When I say steps in the pipeline I mean the entire CI Build pipeline. Step 1: get list of flows from the flows folder and store in a text file. Step 2: install dependencies for prefect Step 3: login to prefect cloud Step 4: deploy each of the flows
Similar to what you have in the GitHub actions script you shared, although I don't need to authenticate with GCP before the deployment when I'm in Cloud Build
But yeah, the challenge I'm facing is that I do the steps as listed above, step 3 complains that prefect isn't installed in the Ubuntu or python images. I can't use a docker image directly in Build, I have to use an existing "builder" or submit my own build via Dockerfile to Google Container Repository and then use that image, which is what I've tried but get the timeout error.
a
Can you share your approach step by step e.g. in a Markdown Gist with screenshots showing what you did? Perhaps this way I could maybe even reproduce, even though I've never used cloud build, or at least see a bigger picture Also you can try using Prefect base image as shown above
d
Okay, I just tried to change to the Prefect base image instead and it still produces the same error message. The Dockerfile is now as simple as: 
FROM prefecthq/prefect:2.0b6-python3.9
And it works correctly, because 
prefect version
is returning the correct values.
I'll try and put everything together into a Gist for you to see if you can reproduce it yourself.
But currently, the extent of the 
cloudbuild.yaml
file is:
Copy code
steps:
- id: 'Generate full list of flows/DeploymentSpecs'
  name: 'ubuntu'
  entrypoint: 'printf'
  args:
    - "'%s\n' workspace/flows/* > flow_list.txt"

- id: 'Login to Prefect Cloud 2.0'
  name: '<http://gcr.io/{project-name}/{image_name}|gcr.io/{project-name}/{image_name}>'
  args: ['prefect', 'cloud', 'login', '--key', '{insert_key}', '--workspace', '{insert_workspace}']
Another update, if I add the 
prefect cloud login
command to the Dockerfile, it successfully logs in. I'm going to see whether the deploy command works as expected when the authorisation is handled in the image. It feels like it's not the correct way of doing it, but if it works then at least I've found a workaround.
Interestingly when I do this, I get the following which says that it is using FileStorage as expected, but I don't see it within the Prefect Cloud 2.0 Workspace...
Copy code
Step #2 - "Create deployment": Loading deployment specifications from python script at 
Step #2 - "Create deployment": 'flows/second_flow.py'...
Step #2 - "Create deployment": Creating deployment 'second-deployment' for flow 'main-flow'...
Step #2 - "Create deployment": Deploying flow script from '/workspace/flows/second_flow.py' using File 
Step #2 - "Create deployment": Storage...
Step #2 - "Create deployment": Created deployment 'main-flow/second-deployment'.
Step #2 - "Create deployment": View your new deployment with: 
Step #2 - "Create deployment": 
Step #2 - "Create deployment":     prefect deployment inspect 'main-flow/second-deployment'
Step #2 - "Create deployment": Created 1 deployments!
Finished Step #2 - "Create deployment"
2022/06/10 17:58:10 Step Step #2 - "Create deployment" finished
2022/06/10 17:58:11 status changed to "DONE"
a
This Gist is amazing, thank you so much! What I meant was that you try to do this instead:
Copy code
steps:
- id: 'Generate full list of flows/DeploymentSpecs'
  name: 'ubuntu'
  entrypoint: 'printf'
  args:
    - "'%s\n' workspace/flows/* > flow_list.txt"

- id: 'Login to Prefect Cloud 2.0'
  name: 'prefecthq/prefect:2.0b6-python3.9'
  args: ['prefect', 'cloud', 'login', '--key', '{insert_key}', '--workspace', '{insert_workspace}']
and great that you at least have a workaround
d
Tried doing as you suggested by using the image directly and again I get the exact same timeout error. Very puzzling.
And unfortunately, I haven't quite found a workaround because even though I can login to the workspace with the method I've used, when I run the deployment (which succeeds, and says it's using FileStorage), the deployment doesn't show up in Prefect Cloud... 🤔
a
I honestly don't know why you can't login from Cloud Build CI - can you try a similar pipeline from GitHub actions as I originally showed? This way, you can confirm whether this is an issue with the code Build setup - I can't replicate your issue on GHA so it may be something with the code build or docker setup
d
I'm going to post in the Google Slack to see if I can get any further from that avenue. Will report back. Thanks for all your help Anna!
🙌 1
a
Great idea!
d
Got it working perfectly with GitHub actions 😎 There's also another mechanism you can use for authentication with GCP that doesn't involve uploading a service account file (better practice). You can read about it here: https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions And the final actions script I have looks like this:
Copy code
name: CI
on:
  push:
    branches: [ main ]
jobs:
  list-flows:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v3
      - id: set-matrix
        run: echo "::set-output name=matrix::$(ls flows/*.py | jq -R -s -c 'split("\n")[:-1]')"
  deploy:
    needs: list-flows
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    strategy:
      matrix:
        flows: ${{ fromJson(needs.list-flows.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.9
        uses: actions/setup-python@v3
        with:
          python-version: 3.9
      - name: Dev dependencies
        run: |
          pip install gcsfs
          pip install -U "prefect>=2.0b"
      - name: Login to Prefect Cloud 2.0
        run: prefect cloud login --key ${{ secrets.PREFECT_API_KEY }} --workspace 'danilodrobacnzytecouk/nzyte-prefect'
      - name: Validate Prefect version
        run: prefect version
      - name: Authenticate to GCP
        uses: 'google-github-actions/auth@v0.7.1'
        with:
          workload_identity_provider: 'projects/170961541935/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
          service_account: '<mailto:prefect-service-account@nzyte-modern-data-stack.iam.gserviceaccount.com|prefect-service-account@nzyte-modern-data-stack.iam.gserviceaccount.com>'
      - name: Create deployment
        run: prefect deployment create ${{ matrix.flows }}
💯 1
🙌 1
a
This is awesome, thanks so much for sharing! I read about it, but then I considered many users already have a JSON file with SA permissions so I assume this is an easier way, but yeah definitely IAM roles ftw 🙌
🙊 1
d
Ahh not so great news when using FileStorage. 
The bq and gsutil tools do not currently support Workload Identity Federation! You will need to use traditional service account key authentication for now.
So looks like you still need to use a service account for authentication when interfacing with GCS or BQ. Typical, but still one to look out for when they update it!
a
wow, this is surprising, thanks for giving this update!
I honestly don't know why GCS makes permissioning so hard, it's so much easier on AWS
d
I've never used AWS but it's definitely a more mature product. Workload Identity is a new (as far as I can tell) feature in GCP and 
bq
and 
gsutil
are legacy packages, I guess they just haven't gotten round to updating the codebase, although you would think that should be part of the rollout of the new product itself so people don't run in to issues like this!
👍 1
a
wow, I'm learning a lot from you here. I thought workload identity is basically IAM roles under the hood but seems I was wrong thanks!
7 Views
Open in Slack
PreviousNext
Powered by