https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-gcp
  • b

    Bianca Hoch

    11/01/2022, 2:17 PM
    @Bianca Hoch has joined the channel
  • j

    Jeff Hale

    11/01/2022, 2:17 PM
    @Jeff Hale has joined the channel
  • r

    Ryan Peden

    11/01/2022, 2:17 PM
    @Ryan Peden has joined the channel
  • k

    Khuyen Tran

    11/01/2022, 2:17 PM
    @Khuyen Tran has joined the channel
  • o

    Orbit

    11/01/2022, 2:54 PM
    @Orbit has joined the channel
  • l

    Linen

    11/01/2022, 2:54 PM
    @Linen has joined the channel
  • d

    Discourse Integration

    11/01/2022, 2:54 PM
    @Discourse Integration has joined the channel
  • n

    Nelson Griffiths

    11/01/2022, 3:03 PM
    @Nelson Griffiths has joined the channel
  • k

    Kevin Grismore

    11/01/2022, 3:07 PM
    @Kevin Grismore has joined the channel
  • r

    Rob Freedy

    11/01/2022, 3:09 PM
    @Rob Freedy has joined the channel
  • j

    Jamie Zieziula

    11/01/2022, 3:17 PM
    @Jamie Zieziula has joined the channel
  • n

    Nate

    11/01/2022, 3:30 PM
    @Nate has joined the channel
  • j

    James Sopkin

    11/01/2022, 3:34 PM
    @James Sopkin has joined the channel
  • o

    Owen McMahon

    11/01/2022, 4:22 PM
    @Owen McMahon has joined the channel
  • a

    Andrew Huang

    11/01/2022, 4:26 PM
    @Andrew Huang has joined the channel
  • j

    Josh

    11/01/2022, 4:35 PM
    @Josh has joined the channel
  • a

    Anthony Head

    11/01/2022, 4:44 PM
    @Anthony Head has joined the channel
  • k

    Kalise Richmond

    11/03/2022, 5:10 PM
    cross-posting: @Ryan Peden has a new article out :gcp: https://prefect-community.slack.com/archives/CL09KTZPX/p1667486942247079
    πŸ™Œ 4
    :gratitude-thank-you: 1
    πŸ‘€ 1
  • e

    Eden

    11/16/2022, 9:06 AM
    Hi gcp ch, After reading many guidelines, I wrote 101 tutorial to help ppl want to use Prefect through GKE. Wish ppl could have fun with Prefect/GKE/dbt. CLICK_ME πŸ™πŸ»
    πŸ™Œ 6
    πŸ‘ 4
    :gratitude-thank-you: 3
    :party-parrot: 1
    ⭐ 2
    a
    • 2
    • 1
  • n

    Nelson Griffiths

    12/05/2022, 7:33 PM
    I am running into some errors today with pushing flows to my cloud storage bucket when applying deployments. I updated the most recent version of both
    prefect
    and
    prefect-gcp
    βœ… 1
    r
    • 2
    • 5
  • s

    Steven Trimboli

    12/07/2022, 6:09 PM
    Hi - I have had trouble passing credentials to validate bigquery for some time now. I pasted the json into a GCP Credentials block Then I try validating in this way:
    from prefect_gcp import GcpCredentials
    
    gcp_credentials_block = GcpCredentials.load("bq-credentials")
    bigquery_client = bigquery.Client(credentials=gcp_credentials_block)
    I receive this error message: "ValueError: This library only supports credentials from google-auth-library-python. See https://google-auth.readthedocs.io/en/latest/ for help on authentication with this library." How can I pass credentials to GCP successfully?
    βœ… 1
    k
    a
    • 3
    • 13
  • y

    Yoanis Gil

    12/07/2022, 6:15 PM
    Hi there. Is there some documentation I can read on running prefect on on spot instances (GCP)? Kinda curious if that something that will work and if so, what setup is require.
    m
    • 2
    • 1
  • n

    Nelson Griffiths

    12/14/2022, 8:48 PM
    This warning started popping up when I run flows locally for testing. It doesn't seem to be interfering with my flows at all, but not sure if this is something I should worry about?
    Warning!  Failed to load collection 'prefect_gcp_aiplatform': ModuleNotFoundError: No module named 'google.cloud.aiplatform'
    βœ… 1
    k
    j
    • 3
    • 3
  • n

    Nelson Griffiths

    12/14/2022, 10:42 PM
    I started getting this error when trying to run a deployment for a cloud run job today:
    Submission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
    Not quite sure what it means?
    βœ… 1
    • 1
    • 1
  • m

    Miguel Moncada

    01/04/2023, 11:36 AM
    πŸ‘‹ Hi, I'm encountering the same error as described above, incognito did not help unfortunately.
    Submission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
    Probably it's due to a misconfiguration on the deployment python file (or maybe the infrastructure block itself?), adding more info to the thread πŸ€”
    βœ… 1
    j
    n
    • 3
    • 15
  • a

    anurag ambuja

    01/05/2023, 10:50 AM
    I am quite new to Prefect and I tried running prefect using cloudrun but the job is not getting triggered at cloudrun. Deployment is working fine. But flow runs are getting into "Late" state with no logs. Not able to debug due to no logs. Any suggestions ?
    βœ… 1
    m
    • 2
    • 5
  • m

    Miguel Moncada

    01/10/2023, 8:51 AM
    πŸ‘‹ Hi, I'm probably (surely) missing something very simple, I'm currently not able to load a bucket from a GCS block using
    GcsBucket.load
    classmethod following the reference here.
    ValueError: Unable to find block document named tam-2759-bucket for block type gcs-bucket
    More details in the thread 🧡
    βœ… 1
    a
    • 2
    • 8
  • g

    Giuliano Mega

    01/19/2023, 1:09 PM
    Hello there folks! I'm a prefect cloud user running agents on GKE autopilot. I've recently updated my prefect agent from 2.7.0 to 2.7.8 to be able to deal with https://github.com/PrefectHQ/prefect/issues/7442 (was getting thousands of those every day), and am now struggling with an issue with the new agent, namely https://github.com/PrefectHQ/prefect/issues/7842 (permission denied while trying to read the
    kube-system
    namespace). As per the workaround in the bug, I've updated my existing staging agent role so that it can read namespaces:
    ➜ ~ kubectl describe role prefect-staging-agent 
    Name:         prefect-staging-agent
    Labels:       <none>
    Annotations:  <none>
    PolicyRule:
      Resources    Non-Resource URLs  Resource Names  Verbs
      ---------    -----------------  --------------  -----
      jobs.batch   []                 []              [list watch create update patch get delete]
      namespaces   []                 []              [list watch get]
      pods/log     []                 []              [list watch get]
      pods/status  []                 []              [list watch get]
      pods         []                 []              [list watch get]
    to no avail. Still getting
    "Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden"
    Any ideas of what might I be missing? Guess my main concern is that kube-system might be off limits for autopilot but couldn't find anything saying you can't read stuff from it.
    βœ… 1
    • 1
    • 3
  • j

    Jack

    01/22/2023, 8:55 PM
    Hello Prefect-GCP team! I have been fortunate to have used great Prefect-GCP/Prefect-AWS repos in the past quarter, and would love to try to contribute to the GCP repo during my free time. Who would I reach out to about possibly contributing?
    πŸ™Œ 2
    βœ… 1
    a
    • 2
    • 2
  • a

    Anthony Harris

    01/31/2023, 7:10 PM
    Hello! We have deployed an Orion agent to GKE autopilot, and we are able to successfully execute flows from the Cloud UI. We're currently investigating whether it is possible to use Vertical Pod Autoscaler to optimize the resources utilized by the pods that are deployed by the Orion agent. From what we can tell online, the VPA is pretty much exclusively pointed at a Deployment object, rather than a Job object - does it make sense to point the VPA at the Orion agent? From the linked documentation, a
    targetRef
    is a:
    Reference to the controller that manages the set of Pods for the autoscaler to control, for example, a Deployment or a StatefulSet. You can point a
    VerticalPodAutoscaler
    at any controller that has a Scale subresource. Typically, the
    VerticalPodAutoscaler
    retrieves the Pod set from the controller's ScaleStatus.
    Any guidance is much appreciated!
    • 1
    • 1
Powered by Linen
Title
a

Anthony Harris

01/31/2023, 7:10 PM
Hello! We have deployed an Orion agent to GKE autopilot, and we are able to successfully execute flows from the Cloud UI. We're currently investigating whether it is possible to use Vertical Pod Autoscaler to optimize the resources utilized by the pods that are deployed by the Orion agent. From what we can tell online, the VPA is pretty much exclusively pointed at a Deployment object, rather than a Job object - does it make sense to point the VPA at the Orion agent? From the linked documentation, a
targetRef
is a:
Reference to the controller that manages the set of Pods for the autoscaler to control, for example, a Deployment or a StatefulSet. You can point a
VerticalPodAutoscaler
at any controller that has a Scale subresource. Typically, the
VerticalPodAutoscaler
retrieves the Pod set from the controller's ScaleStatus.
Any guidance is much appreciated!
Figured this out, btw - the answer is no, the VPA does not apply to jobs created by the Prefect Agent. We're opted to adjust our process to test run a deployment after instantiation, monitor the resource utilization during the test run, and then subsequently adjust the resource requests defined in the KubernetesJob object and redeploy with the new, more optimized resource requests
View count: 1