Anthony Head
11/01/2022, 4:44 PMKalise Richmond
11/03/2022, 5:10 PMEden
11/16/2022, 9:06 AMNelson Griffiths
12/05/2022, 7:33 PMprefect
and prefect-gcp
Steven Trimboli
12/07/2022, 6:09 PMfrom prefect_gcp import GcpCredentials
gcp_credentials_block = GcpCredentials.load("bq-credentials")
bigquery_client = bigquery.Client(credentials=gcp_credentials_block)
I receive this error message: "ValueError: This library only supports credentials from google-auth-library-python. See https://google-auth.readthedocs.io/en/latest/ for help on authentication with this library."
How can I pass credentials to GCP successfully?Yoanis Gil
12/07/2022, 6:15 PMNelson Griffiths
12/14/2022, 8:48 PMWarning! Failed to load collection 'prefect_gcp_aiplatform': ModuleNotFoundError: No module named 'google.cloud.aiplatform'
Nelson Griffiths
12/14/2022, 10:42 PMSubmission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
Not quite sure what it means?Miguel Moncada
01/04/2023, 11:36 AMSubmission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
Probably it's due to a misconfiguration on the deployment python file (or maybe the infrastructure block itself?), adding more info to the thread π€anurag ambuja
01/05/2023, 10:50 AMMiguel Moncada
01/10/2023, 8:51 AMGcsBucket.load
classmethod following the reference here.
ValueError: Unable to find block document named tam-2759-bucket for block type gcs-bucket
More details in the thread π§΅Giuliano Mega
01/19/2023, 1:09 PMkube-system
namespace).
As per the workaround in the bug, I've updated my existing staging agent role so that it can read namespaces:
β ~ kubectl describe role prefect-staging-agent
Name: prefect-staging-agent
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
jobs.batch [] [] [list watch create update patch get delete]
namespaces [] [] [list watch get]
pods/log [] [] [list watch get]
pods/status [] [] [list watch get]
pods [] [] [list watch get]
to no avail. Still getting "Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden"
Any ideas of what might I be missing? Guess my main concern is that kube-system might be off limits for autopilot but couldn't find anything saying you can't read stuff from it.Jack
01/22/2023, 8:55 PMAnthony Harris
01/31/2023, 7:10 PMtargetRef
is a:
Reference to the controller that manages the set of Pods for the autoscaler to control, for example, a Deployment or a StatefulSet. You can point aAny guidance is much appreciated!at any controller that has a Scale subresource. Typically, theVerticalPodAutoscaler
retrieves the Pod set from the controller's ScaleStatus.VerticalPodAutoscaler
Hristo Stefanov
03/09/2023, 9:23 PMMiguel Moncada
03/29/2023, 4:16 PM2.8.7
but I'm getting this error on the last step, which is to register the blocks:
executor failed running [/bin/sh -c prefect block register -m prefect_gcp]: exit code: 1
ERROR: Service 'agent' failed to build : Build failed
Have you seen this error before?Miguel Moncada
03/31/2023, 1:29 PMGcsBucket
to delete a blob, or do I need to directly make use of Google's client? π€
I couldn't find any after checking here.
Thanks a lot!Miguel Moncada
04/05/2023, 5:48 PMSubmission failed. AttributeError: 'Resource' object has no attribute 'jobs'
I'm using prefect_gcp
version 0.3.0
in my agent, and targetting Prefect Cloud. Has anyone seen similar behavior?Eric Ma
04/07/2023, 5:57 AMserviceAccountName
in the YAML creation, it is defaulting to a generic read-only Compute Engine service account.
Do you have any solution on how I can set the default serviceAccountName to use the same service account that is provided in GCP Credential Block?
Thank you in advance for any help here
https://cloud.google.com/run/docs/securing/service-identity#gcloud
By default, Cloud Run revisions and jobs execute as the Compute Engine default service account. The Compute Engine default service account has the Project Editor IAM role which grants read and write permissions on all resources in your Google Cloud project.
Lucas Zago
05/02/2023, 6:49 PMMatt Delacour
05/05/2023, 1:54 PMMatt Delacour
05/05/2023, 2:00 PMMatt Delacour
05/10/2023, 1:21 PMjuandavidlozano
05/10/2023, 11:12 PMupload_from_path
on my code you will see that I am passing the same variable path as the from_path
and the to_path
but for some reason prefect changes the structure of the to_path
variable, here is the code I have that builds the path:
@task()
def write_local(df: pd.DataFrame, color: str, dataset_file: str) -> Path:
"""Write DataFrame out locally as parquet file"""
Path(f"data/{color}").mkdir(parents=True, exist_ok=True)
path = Path(f"data/{color}/{dataset_file}.parquet")
df.to_parquet(path, compression="gzip")
return path
@task
def write_gcs(path: Path) -> None:
"""Upload local parquet file to GCS"""
gcs_block = GcsBucket.load("zoom-gcs")
gcs_block.upload_from_path(from_path=path, to_path=path)
return
you can see in the second task write_gcs
both of the paths are the same variable called path
and that is just a path structure that has originally this value: 'data/yellow/yellow_tripdata_2021-01.parquet'
.
The prefect flows runs, but after it runs, in the details of the flow we can see on the first picture I am attaching it changed the text structure of the path for GCS to: 'data\\yellow\\yellow_tripdata_2021-01.parquet'
, no idea why this is happening and because of this you can see in the picture 2 that it saves the file with that weird name instead of creating the folders in GCS, any help on maybe why this is happening?Nelson Griffiths
05/11/2023, 3:00 PMprefect-gcp
credentials blocks.
When I try to pass my credentials in via a block I get AttributeError: 'str' object has no attribute 'keys'
Ill share the full traceback in the thread. But it seems like it is having a hard time converting my credentials block into actual credentials.Francisco
05/22/2023, 2:41 PMprefect-gcp
cloud run job, and stochastically receive the following error in some flow runs:
KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
With a retry then the error does not appear. After a random number of executions it appears again and after a retry we recover that execution.
Has anyone experienced something similar or know what it could be?Aaron Gonzalez
06/05/2023, 4:38 PMPREFECT_API_URL
env var to the external IP address of my vm instance (not the nicest solution but I can clean this up later) and then I can immediately push my blocks and deployments to the server! π
The problem comes from when I lauch a run of one of my test deployments that use a Cloud Run Job. I never get out of the "Pending" status in the Prefect UI. Pretty sure it is something with my networking and probably not an actual issue with Prefect, but I can't seem to make any progress. I've tried setting up Serverless VPC Connections to allow the Cloud Run Job to be able to connect to the vm that has the server running, but maybe I've set it up incorrectly π€·.
Anyone else ever managed to get a cloud run job deployment to work with a vm instance hosted prefect server?Heliya Hasani
06/08/2023, 1:23 PMJack
07/28/2023, 4:52 AMEric Ma
09/15/2023, 2:48 PM