Has anybody come across an error when using `prefect cloud l

Question

Has anybody come across an error when using `prefect cloud login key key workspace workspace ` related to timeouts I m trying to run the command in a CI CD pipeline through Google Cloud Build but it fails due to a timeout and I have very little information in the error

Anna Geller · Answer

There is one caveat you need to specify your account not only the workspace name since the workspace name is not unique without the account name

Anna Geller · Answer

e g my account is annaprefect and workspace is prod so I would need to use ```prefect cloud login k API KEY w annaprefect prod ```

Danilo Drobac · Answer

Yep I m doing that

Anna Geller · Answer

Can you recreate the API key and ensure it won t expire too soon definitely worth replacing the API key

Danilo Drobac · Answer

I ve tried the command locally and it works perfectly just tried it now but the CI CD still times out

Anna Geller · Answer

looks like your CI is using older Prefect 2 0 version which has older now incompatible API version

Anna Geller · Answer

try using 2 0b6 from your CI

Danilo Drobac · Answer

This is what I m getting from my CI builder that I created in a Dockerfile ```Step 2 Prefect Version Version 2 0b6 Step 2 Prefect Version API version 0 5 0 Step 2 Prefect Version Python version 3 9 13 Step 2 Prefect Version Git commit 502ee008 Step 2 Prefect Version Built Tue Jun 7 2022 4 58 PM Step 2 Prefect Version OS Arch linux x86 64 Step 2 Prefect Version Profile default Step 2 Prefect Version Server type ephemeral Step 2 Prefect Version Server Step 2 Prefect Version Database sqlite Step 2 Prefect Version SQLite version 3 34 1```

Anna Geller · Answer

is this a GitHub action

Danilo Drobac · Answer

No it s using Google Cloud Build

Anna Geller · Answer

what s the error

Danilo Drobac · Answer

```Starting Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 Already have image with digest Step 2 Login to Prefect Cloud 2 0 Traceback most recent call last Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages anyio core sockets py line 186 in connect tcp Step 2 Login to Prefect Cloud 2 0 addr obj = ip address remote host Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 ipaddress py line 53 in ip address Step 2 Login to Prefect Cloud 2 0 raise ValueError f address r does not appear to be an IPv4 or IPv6 address Step 2 Login to Prefect Cloud 2 0 ValueError does not appear to be an IPv4 or IPv6 address Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 During handling of the above exception another exception occurred Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 Traceback most recent call last Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore backends asyncio py line 109 in connect tcp Step 2 Login to Prefect Cloud 2 0 stream anyio abc ByteStream = await anyio connect tcp Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages anyio core sockets py line 189 in connect tcp Step 2 Login to Prefect Cloud 2 0 gai res = await getaddrinfo Step 2 Login to Prefect Cloud 2 0 asyncio exceptions CancelledError Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 During handling of the above exception another exception occurred Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 Traceback most recent call last Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore exceptions py line 8 in map exceptions Step 2 Login to Prefect Cloud 2 0 yield Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore backends asyncio py line 109 in connect tcp Step 2 Login to Prefect Cloud 2 0 stream anyio abc ByteStream = await anyio connect tcp Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages anyio core tasks py line 118 in exit Step 2 Login to Prefect Cloud 2 0 raise TimeoutError Step 2 Login to Prefect Cloud 2 0 TimeoutError Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 During handling of the above exception another exception occurred Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 Traceback most recent call last Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx transports default py line 60 in map httpcore exceptions Step 2 Login to Prefect Cloud 2 0 yield Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx transports default py line 353 in handle async request Step 2 Login to Prefect Cloud 2 0 resp = await self pool handle async request req Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore async connection pool py line 253 in handle async request Step 2 Login to Prefect Cloud 2 0 raise exc Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore async connection pool py line 237 in handle async request Step 2 Login to Prefect Cloud 2 0 response = await connection handle async request request Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore async connection py line 86 in handle async request Step 2 Login to Prefect Cloud 2 0 raise exc Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore async connection py line 63 in handle async request Step 2 Login to Prefect Cloud 2 0 stream = await self connect request Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore async connection py line 111 in connect Step 2 Login to Prefect Cloud 2 0 stream = await self network backend connect tcp kwargs Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore backends auto py line 29 in connect tcp Step 2 Login to Prefect Cloud 2 0 return await self backend connect tcp Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore backends asyncio py line 109 in connect tcp Step 2 Login to Prefect Cloud 2 0 stream anyio abc ByteStream = await anyio connect tcp Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 contextlib py line 137 in exit Step 2 Login to Prefect Cloud 2 0 self gen throw typ value traceback Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpcore exceptions py line 12 in map exceptions Step 2 Login to Prefect Cloud 2 0 raise to exc exc Step 2 Login to Prefect Cloud 2 0 httpcore ConnectTimeout Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 The above exception was the direct cause of the following exception Step 2 Login to Prefect Cloud 2 0 Step 2 Login to Prefect Cloud 2 0 Traceback most recent call last Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages prefect cli utilities py line 44 in wrapper Step 2 Login to Prefect Cloud 2 0 return fn args kwargs Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages prefect utilities asyncio py line 122 in wrapper Step 2 Login to Prefect Cloud 2 0 return run async in new loop async fn args kwargs Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages prefect utilities asyncio py line 69 in run async in new loop Step 2 Login to Prefect Cloud 2 0 return anyio run partial fn args kwargs Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages anyio core eventloop py line 70 in run Step 2 Login to Prefect Cloud 2 0 return asynclib run func args backend options Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages anyio backends asyncio py line 292 in run Step 2 Login to Prefect Cloud 2 0 return native run wrapper debug=debug Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 asyncio runners py line 44 in run Step 2 Login to Prefect Cloud 2 0 return loop run until complete main Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 asyncio base events py line 647 in run until complete Step 2 Login to Prefect Cloud 2 0 return future result Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages anyio backends asyncio py line 287 in wrapper Step 2 Login to Prefect Cloud 2 0 return await func args Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages prefect cli cloud py line 204 in login Step 2 Login to Prefect Cloud 2 0 workspaces = await client read workspaces Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages prefect cli cloud py line 82 in read workspaces Step 2 Login to Prefect Cloud 2 0 return await self get me workspaces Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages prefect cli cloud py line 102 in get Step 2 Login to Prefect Cloud 2 0 res = await self client get route kwargs Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx client py line 1751 in get Step 2 Login to Prefect Cloud 2 0 return await self request Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx client py line 1527 in request Step 2 Login to Prefect Cloud 2 0 return await self send request auth=auth follow redirects=follow redirects Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx client py line 1614 in send Step 2 Login to Prefect Cloud 2 0 response = await self send handling auth Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx client py line 1642 in send handling auth Step 2 Login to Prefect Cloud 2 0 response = await self send handling redirects Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx client py line 1679 in send handling redirects Step 2 Login to Prefect Cloud 2 0 response = await self send single request request Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx client py line 1716 in send single request Step 2 Login to Prefect Cloud 2 0 response = await transport handle async request request Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx transports default py line 353 in handle async request Step 2 Login to Prefect Cloud 2 0 resp = await self pool handle async request req Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 contextlib py line 137 in exit Step 2 Login to Prefect Cloud 2 0 self gen throw typ value traceback Step 2 Login to Prefect Cloud 2 0 File usr local lib python3 9 site packages httpx transports default py line 77 in map httpcore exceptions Step 2 Login to Prefect Cloud 2 0 raise mapped exc message from exc Step 2 Login to Prefect Cloud 2 0 httpx ConnectTimeout Step 2 Login to Prefect Cloud 2 0 An exception occurred ```

Anna Geller · Answer

can you share Dockerfile you used for <http gcr io nzyte modern data stack prefect builder|gcr io nzyte modern data stack prefect builder>

Anna Geller · Answer

in general you don t need to run this in Docker we run all tests on Ubuntu so this might be easier to set up

Anna Geller · Answer

you would only need to install this in your CI ```pip install gcsfs pip install U prefect gt =2 0b ``` all other dependencies from your Docker image can be used at runtime in your DockerFlowRunner or KubernetesFlowRunner but they don t need to be available during deployment creation process

Anna Geller · Answer

I think since you mentioned the same cloud login command login works locally my feeling is there is some weird network issue given that you run this in a Docker container then Google Cloud Build may itself run in a Docker container and it s somehow causes issues it s worth trying to start CI with the latest Ubuntu version and install Prefect as part of CI before creating deployment

Danilo Drobac · Answer

The Dockerfile is based on python 3 9 and just installs those 2 dependencies I was struggling to get the prefect installation to persist through different steps in the build pipeline so I first tried to do it all in a single step that started from Python3 9 installed the dependencies and then logged into the workspace but that gave the same timeout error I agree with you that it may be something network related but the lack of error message makes it difficult I m going to try and ping the host <http api beta prefect io|api beta prefect io> to see if it succeeds that should be a good first step

Anna Geller · Answer

Yes for sure What do you mean by multiple steps in the pipeline all you need is to login to cloud and create a deployment

Anna Geller · Answer

curious to know what your CI consists of

Anna Geller · Answer

btw you can also use Prefect 2 0 base image instead of Python 3 9 just something you may consider ```FROM prefecthq prefect 2 0b6 python3 9```

Danilo Drobac · Answer

When I say steps in the pipeline I mean the entire CI Build pipeline Step 1 get list of flows from the flows folder and store in a text file Step 2 install dependencies for prefect Step 3 login to prefect cloud Step 4 deploy each of the flows

Danilo Drobac · Answer

Similar to what you have in the GitHub actions script you shared although I don t need to authenticate with GCP before the deployment when I m in Cloud Build

Danilo Drobac · Answer

But yeah the challenge I m facing is that I do the steps as listed above step 3 complains that prefect isn t installed in the Ubuntu or python images I can t use a docker image directly in Build I have to use an existing builder or submit my own build via Dockerfile to Google Container Repository and then use that image which is what I ve tried but get the timeout error

Anna Geller · Answer

Can you share your approach step by step e g in a Markdown Gist with screenshots showing what you did Perhaps this way I could maybe even reproduce even though I ve never used cloud build or at least see a bigger picture Also you can try using Prefect base image as shown above

Danilo Drobac · Answer

Okay I just tried to change to the Prefect base image instead and it still produces the same error message The Dockerfile is now as simple as `FROM prefecthq prefect 2 0b6 python3 9` And it works correctly because `prefect version` is returning the correct values

Danilo Drobac · Answer

I ll try and put everything together into a Gist for you to see if you can reproduce it yourself

Danilo Drobac · Answer

But currently the extent of the `cloudbuild yaml` file is ```steps id Generate full list of flows DeploymentSpecs name ubuntu entrypoint printf args s n workspace flows gt flow list txt id Login to Prefect Cloud 2 0 name <http gcr io project name image name |gcr io project name image name > args prefect cloud login key insert key workspace insert workspace ```

Danilo Drobac · Answer

Here s the Gist I Just put together <https gist github com danilo nzyte 653d788e8f4a3b91ef72c2bc722fde51>

Danilo Drobac · Answer

Another update if I add the `prefect cloud login` command to the Dockerfile it successfully logs in I m going to see whether the deploy command works as expected when the authorisation is handled in the image It feels like it s not the correct way of doing it but if it works then at least I ve found a workaround

Danilo Drobac · Answer

Interestingly when I do this I get the following which says that it is using FileStorage as expected but I don t see it within the Prefect Cloud 2 0 Workspace ```Step 2 Create deployment Loading deployment specifications from python script at Step 2 Create deployment flows second flow py Step 2 Create deployment Creating deployment second deployment for flow main flow Step 2 Create deployment Deploying flow script from workspace flows second flow py using File Step 2 Create deployment Storage Step 2 Create deployment Created deployment main flow second deployment Step 2 Create deployment View your new deployment with Step 2 Create deployment Step 2 Create deployment prefect deployment inspect main flow second deployment Step 2 Create deployment Created 1 deployments Finished Step 2 Create deployment 2022 06 10 17 58 10 Step Step 2 Create deployment finished 2022 06 10 17 58 11 status changed to DONE ```

Anna Geller · Answer

This Gist is amazing thank you so much What I meant was that you try to do this instead ```steps id Generate full list of flows DeploymentSpecs name ubuntu entrypoint printf args s n workspace flows gt flow list txt id Login to Prefect Cloud 2 0 name prefecthq prefect 2 0b6 python3 9 args prefect cloud login key insert key workspace insert workspace ```

Anna Geller · Answer

and great that you at least have a workaround

Danilo Drobac · Answer

Tried doing as you suggested by using the image directly and again I get the exact same timeout error Very puzzling

Danilo Drobac · Answer

And unfortunately I haven t quite found a workaround because even though I can login to the workspace with the method I ve used when I run the deployment which succeeds and says it s using FileStorage the deployment doesn t show up in Prefect Cloud thinking face

Anna Geller · Answer

I honestly don t know why you can t login from Cloud Build CI can you try a similar pipeline from GitHub actions as I originally showed This way you can confirm whether this is an issue with the code Build setup I can t replicate your issue on GHA so it may be something with the code build or docker setup

Danilo Drobac · Answer

I m going to post in the Google Slack to see if I can get any further from that avenue Will report back Thanks for all your help Anna

Anna Geller · Answer

Great idea

Danilo Drobac · Answer

Got it working perfectly with GitHub actions sunglasses There s also another mechanism you can use for authentication with GCP that doesn t involve uploading a service account file better practice You can read about it here And the final actions script I have looks like this ```name CI on push branches main jobs list flows runs on ubuntu latest outputs matrix $ steps set matrix outputs matrix steps uses actions checkout v3 id set matrix run echo set output name=matrix $ ls flows py | jq R s c split n 1 deploy needs list flows runs on ubuntu latest permissions contents read id token write strategy matrix flows $ fromJson needs list flows outputs matrix steps uses actions checkout v3 name Set up Python 3 9 uses actions setup python v3 with python version 3 9 name Dev dependencies run | pip install gcsfs pip install U prefect gt =2 0b name Login to Prefect Cloud 2 0 run prefect cloud login key $ secrets PREFECT API KEY workspace danilodrobacnzytecouk nzyte prefect name Validate Prefect version run prefect version name Authenticate to GCP uses google github actions auth v0 7 1 with workload identity provider projects 170961541935 locations global workloadIdentityPools github actions pool providers github actions provider service account name Create deployment run prefect deployment create $ matrix flows ```

Anna Geller · Answer

This is awesome thanks so much for sharing I read about it but then I considered many users already have a JSON file with SA permissions so I assume this is an easier way but yeah definitely IAM roles ftw raised hands

Danilo Drobac · Answer

Ahh not so great news when using FileStorage `The bq and gsutil tools do not currently support Workload Identity Federation You will need to use traditional service account key authentication for now ` So looks like you still need to use a service account for authentication when interfacing with GCS or BQ Typical but still one to look out for when they update it

Anna Geller · Answer

wow this is surprising thanks for giving this update

Anna Geller · Answer

I honestly don t know why GCS makes permissioning so hard it s so much easier on AWS

Danilo Drobac · Answer

I ve never used AWS but it s definitely a more mature product Workload Identity is a new as far as I can tell feature in GCP and `bq` and `gsutil` are legacy packages I guess they just haven t gotten round to updating the codebase although you would think that should be part of the rollout of the new product itself so people don t run in to issues like this

Anna Geller · Answer

wow I m learning a lot from you here I thought workload identity is basically IAM roles under the hood but seems I was wrong thanks