https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-gcp
  • k

    Kalise Richmond

    11/01/2022, 2:16 PM
    @Kalise Richmond has joined the channel
    :gcp: 1
  • a

    Anna Geller

    11/01/2022, 2:17 PM
    @Anna Geller has joined the channel
  • m

    Mason Menges

    11/01/2022, 2:17 PM
    @Mason Menges has joined the channel
  • b

    Bianca Hoch

    11/01/2022, 2:17 PM
    @Bianca Hoch has joined the channel
  • j

    Jeff Hale

    11/01/2022, 2:17 PM
    @Jeff Hale has joined the channel
  • r

    Ryan Peden

    11/01/2022, 2:17 PM
    @Ryan Peden has joined the channel
  • k

    Khuyen Tran

    11/01/2022, 2:17 PM
    @Khuyen Tran has joined the channel
  • o

    Orbit

    11/01/2022, 2:54 PM
    @Orbit has joined the channel
  • l

    Linen

    11/01/2022, 2:54 PM
    @Linen has joined the channel
  • d

    Discourse Integration

    11/01/2022, 2:54 PM
    @Discourse Integration has joined the channel
  • n

    Nelson Griffiths

    11/01/2022, 3:03 PM
    @Nelson Griffiths has joined the channel
  • k

    Kevin Grismore

    11/01/2022, 3:07 PM
    @Kevin Grismore has joined the channel
  • r

    Rob Freedy

    11/01/2022, 3:09 PM
    @Rob Freedy has joined the channel
  • j

    Jamie Zieziula

    11/01/2022, 3:17 PM
    @Jamie Zieziula has joined the channel
  • n

    Nate

    11/01/2022, 3:30 PM
    @Nate has joined the channel
  • j

    James Sopkin

    11/01/2022, 3:34 PM
    @James Sopkin has joined the channel
  • o

    Owen McMahon

    11/01/2022, 4:22 PM
    @Owen McMahon has joined the channel
  • a

    Andrew Huang

    11/01/2022, 4:26 PM
    @Andrew Huang has joined the channel
  • j

    Josh

    11/01/2022, 4:35 PM
    @Josh has joined the channel
  • a

    Anthony Head

    11/01/2022, 4:44 PM
    @Anthony Head has joined the channel
  • k

    Kalise Richmond

    11/03/2022, 5:10 PM
    cross-posting: @Ryan Peden has a new article out :gcp: https://prefect-community.slack.com/archives/CL09KTZPX/p1667486942247079
    πŸ™Œ 4
    :gratitude-thank-you: 1
    πŸ‘€ 1
  • e

    Eden

    11/16/2022, 9:06 AM
    Hi gcp ch, After reading many guidelines, I wrote 101 tutorial to help ppl want to use Prefect through GKE. Wish ppl could have fun with Prefect/GKE/dbt. CLICK_ME πŸ™πŸ»
    πŸ™Œ 6
    πŸ‘ 4
    :gratitude-thank-you: 3
    :party-parrot: 1
    ⭐ 2
    a
    1 reply Β· 2 participants
  • n

    Nelson Griffiths

    12/05/2022, 7:33 PM
    I am running into some errors today with pushing flows to my cloud storage bucket when applying deployments. I updated the most recent version of both
    prefect
    and
    prefect-gcp
    βœ… 1
    r
    5 replies Β· 2 participants
  • s

    Steven Trimboli

    12/07/2022, 6:09 PM
    Hi - I have had trouble passing credentials to validate bigquery for some time now. I pasted the json into a GCP Credentials block Then I try validating in this way:
    from prefect_gcp import GcpCredentials
    
    gcp_credentials_block = GcpCredentials.load("bq-credentials")
    bigquery_client = bigquery.Client(credentials=gcp_credentials_block)
    I receive this error message: "ValueError: This library only supports credentials from google-auth-library-python. See https://google-auth.readthedocs.io/en/latest/ for help on authentication with this library." How can I pass credentials to GCP successfully?
    βœ… 1
    k
    a
    13 replies Β· 3 participants
  • y

    Yoanis Gil

    12/07/2022, 6:15 PM
    Hi there. Is there some documentation I can read on running prefect on on spot instances (GCP)? Kinda curious if that something that will work and if so, what setup is require.
    m
    1 reply Β· 2 participants
  • n

    Nelson Griffiths

    12/14/2022, 8:48 PM
    This warning started popping up when I run flows locally for testing. It doesn't seem to be interfering with my flows at all, but not sure if this is something I should worry about?
    Warning!  Failed to load collection 'prefect_gcp_aiplatform': ModuleNotFoundError: No module named 'google.cloud.aiplatform'
    βœ… 1
    k
    j
    3 replies Β· 3 participants
  • n

    Nelson Griffiths

    12/14/2022, 10:42 PM
    I started getting this error when trying to run a deployment for a cloud run job today:
    Submission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
    Not quite sure what it means?
    βœ… 1
    1 reply Β· 1 participant
  • m

    Miguel Moncada

    01/04/2023, 11:36 AM
    πŸ‘‹ Hi, I'm encountering the same error as described above, incognito did not help unfortunately.
    Submission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
    Probably it's due to a misconfiguration on the deployment python file (or maybe the infrastructure block itself?), adding more info to the thread πŸ€”
    βœ… 1
    j
    n
    15 replies Β· 3 participants
  • a

    anurag ambuja

    01/05/2023, 10:50 AM
    I am quite new to Prefect and I tried running prefect using cloudrun but the job is not getting triggered at cloudrun. Deployment is working fine. But flow runs are getting into "Late" state with no logs. Not able to debug due to no logs. Any suggestions ?
    βœ… 1
    m
    5 replies Β· 2 participants
  • m

    Miguel Moncada

    01/10/2023, 8:51 AM
    πŸ‘‹ Hi, I'm probably (surely) missing something very simple, I'm currently not able to load a bucket from a GCS block using
    GcsBucket.load
    classmethod following the reference here.
    ValueError: Unable to find block document named tam-2759-bucket for block type gcs-bucket
    More details in the thread 🧡
    βœ… 1
    a
    8 replies Β· 2 participants
Powered by Linen
Title
m

Miguel Moncada

01/10/2023, 8:51 AM
πŸ‘‹ Hi, I'm probably (surely) missing something very simple, I'm currently not able to load a bucket from a GCS block using
GcsBucket.load
classmethod following the reference here.
ValueError: Unable to find block document named tam-2759-bucket for block type gcs-bucket
More details in the thread 🧡
βœ… 1
Moreover, I'm able to use the bucket using the code snippet offered by the UI (though it is my understanding that
GcsBucket
class has further options and it's the preferred way-to-go)
logger = get_run_logger()
<http://logger.info|logger.info>("Starting flow")
bucket_gcs = GCS.load(GCS_BUCKET_BLOB)
<http://logger.info|logger.info>(f"Bucket {bucket_gcs} loaded")
❯ python tam_2759_flow.py
09:44:22.436 | INFO    | prefect.engine - Created flow run 'tunneling-horse' for flow 'tam_2759_flow'
09:44:23.376 | INFO    | Flow run 'tunneling-horse' - Starting flow
09:44:23.668 | INFO    | Flow run 'tunneling-horse' - Bucket GCS(bucket_path='ide_chile_data_uploads', service_account_info=SecretStr('**********'), project='cartodb-on-gcp-strategic-sol') loaded
Flow:
from prefect import task, flow, get_run_logger
from prefect_gcp.cloud_storage import GcsBucket
from utils.tam_2759.utils import check_is_candidate_file_from_uri
from utils.tam_2759.constants import GCS_BUCKET_BLOB, FILE_MAX_OLD_DAYS


@task(
    name="get_gcs_candidate_upload_files",
    retries=0,
    retry_delay_seconds=1,
    timeout_seconds=5,
)
def get_gcs_candidate_upload_files(
    bucket: GcsBucket, days_period: int = FILE_MAX_OLD_DAYS
) -> list:
    """
    Function to get the list of files to process.
    Args:
        bucket (GcsBucket): GCS bucket object
        days_period (int, optional): Days period to check if a file is valid
            to process. Defaults to FILE_MAX_OLD_DAYS.
    Returns:
        list: List of files to process
    """
    logger = get_run_logger()
    blob_list = bucket.list_blobs()

    if blob_list:
        <http://logger.info|logger.info>(f"Found {len(blob_list)} files in bucket"
                    f"{GcsBucket.bucket}")
    else:
        logger.warning(f"No files found in bucket {GcsBucket.bucket}")

    return [
        check_is_candidate_file_from_uri(blob.name, days_period=days_period)
        for blob in blob_list
    ]


@flow(name="tam_2759_flow")
def main():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Starting flow")

    bucket = GcsBucket.load(GCS_BUCKET_BLOB)
    <http://logger.info|logger.info>(f"Bucket {bucket} loaded")

    candidate_files = get_gcs_candidate_upload_files(bucket=bucket)
    <http://logger.info|logger.info>(candidate_files)


main()
Prefect / Google Cloud versions:
❯ pip freeze | grep -i "prefect\|google"
google-api-core==2.11.0
google-api-python-client==2.70.0
google-auth==2.15.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.8.0
google-cloud==0.34.0
google-cloud-bigquery==3.4.1
google-cloud-core==2.3.2
google-cloud-storage==2.7.0
google-crc32c==1.5.0
google-resumable-media==2.4.0
googleapis-common-protos==1.57.0
prefect==2.7.5
prefect-dask==0.2.2
prefect-gcp==0.2.2
a

alex

01/10/2023, 2:34 PM
Hey @Miguel Moncada! You’ve created a
GCS
block in the UI, but you’re using the
GcsBucket
class from
prefect-gcp
. If you want to load the block you created in the UI you’ll need to use the
GCS
class from
prefect.filesystems
.
m

Miguel Moncada

01/10/2023, 2:35 PM
How can I benefit from a
GcsBucket
class object using the credentials stored in a GCS block?
a

alex

01/10/2023, 2:42 PM
Great question! You can register the blocks from
prefect-gcp
with the command
prefect blocks register -m prefect_gcp
. The blocks in
prefect_gcp
will be available in your UI and available for configuration after that.
:marvin: 1
I can’t think of a use case where you would need both.
m

Miguel Moncada

01/11/2023, 12:26 PM
Thanks @alex it seems that (if I'm not missing something)
GCS Bucket
block offers more functionality than
GCS
, so I'll try to stick to this one. If that was the case, and just out of curiosity, are there any plans on keeping only one of them in the future?
View count: 1