@Anthony Head has joined the channel

cross-posting: @Ryan Peden has a new article out gcp https://prefect-community.slack.com/archives/CL09KTZPX/p1667486942247079

Hi gcp ch, After reading many guidelines, I wrote 101 tutorial to help ppl want to use Prefect through GKE. Wish ppl could have fun with Prefect/GKE/dbt. CLICK_ME 🙏🏻

I am running into some errors today with pushing flows to my cloud storage bucket when applying deployments. I updated the most recent version of both

prefect

and

prefect-gcp

Hi - I have had trouble passing credentials to validate bigquery for some time now. I pasted the json into a GCP Credentials block Then I try validating in this way:

from prefect_gcp import GcpCredentials

gcp_credentials_block = GcpCredentials.load("bq-credentials")
bigquery_client = bigquery.Client(credentials=gcp_credentials_block)

I receive this error message: "ValueError: This library only supports credentials from google-auth-library-python. See https://google-auth.readthedocs.io/en/latest/ for help on authentication with this library." How can I pass credentials to GCP successfully?

Hi there. Is there some documentation I can read on running prefect on on spot instances (GCP)? Kinda curious if that something that will work and if so, what setup is require.

This warning started popping up when I run flows locally for testing. It doesn't seem to be interfering with my flows at all, but not sure if this is something I should worry about?

Warning!  Failed to load collection 'prefect_gcp_aiplatform': ModuleNotFoundError: No module named 'google.cloud.aiplatform'

I started getting this error when trying to run a deployment for a cloud run job today:

Submission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."

Not quite sure what it means?

👋 Hi, I'm encountering the same error as described above, incognito did not help unfortunately.

Submission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."

Probably it's due to a misconfiguration on the deployment python file (or maybe the infrastructure block itself?), adding more info to the thread 🤔

I am quite new to Prefect and I tried running prefect using cloudrun but the job is not getting triggered at cloudrun. Deployment is working fine. But flow runs are getting into "Late" state with no logs. Not able to debug due to no logs. Do I need to start the agent in the container ? Any suggestions ?

👋 Hi, I'm probably (surely) missing something very simple, I'm currently not able to load a bucket from a GCS block using

GcsBucket.load

classmethod following the reference here.

ValueError: Unable to find block document named tam-2759-bucket for block type gcs-bucket

More details in the thread 🧵

Hello there folks! I'm a prefect cloud user running agents on GKE autopilot. I've recently updated my prefect agent from 2.7.0 to 2.7.8 to be able to deal with https://github.com/PrefectHQ/prefect/issues/7442 (was getting thousands of those every day), and am now struggling with an issue with the new agent, namely https://github.com/PrefectHQ/prefect/issues/7842 (permission denied while trying to read the

kube-system

namespace). As per the workaround in the bug, I've updated my existing staging agent role so that it can read namespaces:

➜ ~ kubectl describe role prefect-staging-agent 
Name:         prefect-staging-agent
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources    Non-Resource URLs  Resource Names  Verbs
  ---------    -----------------  --------------  -----
  jobs.batch   []                 []              [list watch create update patch get delete]
  namespaces   []                 []              [list watch get]
  pods/log     []                 []              [list watch get]
  pods/status  []                 []              [list watch get]
  pods         []                 []              [list watch get]

to no avail. Still getting

"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden"

Any ideas of what might I be missing? Guess my main concern is that kube-system might be off limits for autopilot but couldn't find anything saying you can't read stuff from it.

Hello Prefect-GCP team! I have been fortunate to have used great Prefect-GCP/Prefect-AWS repos in the past quarter, and would love to try to contribute to the GCP repo during my free time. Who would I reach out to about possibly contributing?

Hello! We have deployed an Orion agent to GKE autopilot, and we are able to successfully execute flows from the Cloud UI. We're currently investigating whether it is possible to use Vertical Pod Autoscaler to optimize the resources utilized by the pods that are deployed by the Orion agent. From what we can tell online, the VPA is pretty much exclusively pointed at a Deployment object, rather than a Job object - does it make sense to point the VPA at the Orion agent? From the linked documentation, a

targetRef

is a:

Reference to the controller that manages the set of Pods for the autoscaler to control, for example, a Deployment or a StatefulSet. You can point a

VerticalPodAutoscaler

at any controller that has a Scale subresource. Typically, the

VerticalPodAutoscaler

retrieves the Pod set from the controller's ScaleStatus.

Any guidance is much appreciated!

Path is doubled during deployment. Subsequent flow runs fail. In my gcs-bucket block I have specified the folder to be 1. Deployment is successful and the files are uploaded to folder 1/1/ . Flow runs are unsuccessful: Flow could not be retrieved from deployment. Has anyone else had this issue? Thanks in advance. 😃 I have tried to change the folder with and without slash, etc. There is nothing in the documentation nor in Discourse that specifies how the folder should be entered. I think that this might be a bug because I am using only prefect cli and cloud UI.

Hi team! I'm trying to upgrade my agent to use Prefect version

2.8.7

but I'm getting this error on the last step, which is to register the blocks:

executor failed running [/bin/sh -c prefect block register -m prefect_gcp]: exit code: 1
ERROR: Service 'agent' failed to build : Build failed

Have you seen this error before?

Hi! is there any built-in operation for

GcsBucket

to delete a blob, or do I need to directly make use of Google's client? 🤔 I couldn't find any after checking here. Thanks a lot!

Hi! I've been working in local for a little while, I had earlier been able to deploy and run flows using Google Cloud Run, but I'm recently encountering the following error:

Submission failed. AttributeError: 'Resource' object has no attribute 'jobs'

I'm using

prefect_gcp

version

0.3.0

in my agent, and targetting Prefect Cloud. Has anyone seen similar behavior?

Hi all, I am trying to use prefect-gcp for Google Cloud Run, but when the cloud run job is created, it is using a different service account than the gcpcredential json that I provided. Since the library is not specifying the

serviceAccountName

in the YAML creation, it is defaulting to a generic read-only Compute Engine service account. Do you have any solution on how I can set the default serviceAccountName to use the same service account that is provided in GCP Credential Block? Thank you in advance for any help here https://cloud.google.com/run/docs/securing/service-identity#gcloud

By default, Cloud Run revisions and jobs execute as the Compute Engine default service account. The Compute Engine default service account has the Project Editor IAM role which grants read and write permissions on all resources in your Google Cloud project.

Hi guys Have some issues installing prefect in Mac M2, there is some default requirements.txt to share to install it?

👋 How can I open an issue? This does not allow me

☝️ Is anyone else having similar issues with the agents? You can see that the original conversation of the prefect agent dying every other day is starting to be pretty long ... https://github.com/PrefectHQ/prefect/issues/7442#issuecomment-1536268037

Hi @alex any progress on the above issue? It has other consequences like it messes up with the concurrent task feature too. Like if a job died in GCP but Prefect cloud is not aware, it will not decrease the number of "active tasks" for a specific concurrent tag. Which then can just block them all (see this post)

Hi I have an issue when writing to a google cloud storage, when I use the

upload_from_path

on my code you will see that I am passing the same variable path as the

from_path

and the

to_path

but for some reason prefect changes the structure of the

to_path

variable, here is the code I have that builds the path:

@task()
def write_local(df: pd.DataFrame, color: str, dataset_file: str) -> Path:
    """Write DataFrame out locally as parquet file"""
    Path(f"data/{color}").mkdir(parents=True, exist_ok=True)
    path = Path(f"data/{color}/{dataset_file}.parquet")
    df.to_parquet(path, compression="gzip")
    return path


@task
def write_gcs(path: Path) -> None:
    """Upload local parquet file to GCS"""
    gcs_block = GcsBucket.load("zoom-gcs")
    gcs_block.upload_from_path(from_path=path, to_path=path)
    return

you can see in the second task

write_gcs

both of the paths are the same variable called

path

and that is just a path structure that has originally this value:

'data/yellow/yellow_tripdata_2021-01.parquet'

. The prefect flows runs, but after it runs, in the details of the flow we can see on the first picture I am attaching it changed the text structure of the path for GCS to:

'data\\yellow\\yellow_tripdata_2021-01.parquet'

, no idea why this is happening and because of this you can see in the picture 2 that it saves the file with that weird name instead of creating the folders in GCS, any help on maybe why this is happening?

I am running into some issues trying to get prefect projects working with

prefect-gcp

credentials blocks. When I try to pass my credentials in via a block I get

AttributeError: 'str' object has no attribute 'keys'

Ill share the full traceback in the thread. But it seems like it is having a hard time converting my credentials block into actual credentials.

Hi! In our team we run our flows using

prefect-gcp

cloud run job, and stochastically receive the following error in some flow runs:

KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."

With a retry then the error does not appear. After a random number of executions it appears again and after a retry we recover that execution. Has anyone experienced something similar or know what it could be?

Hello everybody! I've been playing around with the idea of self hosting Prefect server on a VM instance using docker compose. The following posts have gotten me pretty much all the way to having a working self-host option: • https://dev.to/calvinqc/the-easiest-docker-docker-compose-setup-on-compute-engine-1op1 • https://medium.com/the-prefect-blog/how-to-self-host-prefect-orion-with-postgres-using-docker-compose-631c41ab8a9f I am able to just set my local

PREFECT_API_URL

env var to the external IP address of my vm instance (not the nicest solution but I can clean this up later) and then I can immediately push my blocks and deployments to the server! 👍 The problem comes from when I lauch a run of one of my test deployments that use a Cloud Run Job. I never get out of the "Pending" status in the Prefect UI. Pretty sure it is something with my networking and probably not an actual issue with Prefect, but I can't seem to make any progress. I've tried setting up Serverless VPC Connections to allow the Cloud Run Job to be able to connect to the vm that has the server running, but maybe I've set it up incorrectly 🤷. Anyone else ever managed to get a cloud run job deployment to work with a vm instance hosted prefect server?

Hi everyone ! I have an issue regarding to GCP. When I trigger docker image in GCP UI as a cloud run job it is successful however when I try to use prefect and trigger a cloud run I can see the job in both cloud run and prefect ui. But it crashes all the time. Log is : File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration.py", line 1634, in read_flow_run response = await self._client.get(f"/flow_runs/{flow_run_id}") I have checked everywhere but couldn't find a solution. I can reach the flow id and name however I dont know what to do with it.

Hello, I am thinking about adding in Cloud Run API v2 support for Cloud Run Jobs in the Prefect-GCP repository. Should I first make and test this setup then I would post issue with PR attached? Or what's the best procedure?

Hello, is there a way to increase the Google Cloud Run Job block timeout beyond 3600 sec? Right now, the max is set to 3600. However it seems like Google supports up to 24 hours https://cloud.google.com/run/docs/configuring/task-timeout