https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • z

    Zheng Cheng

    06/25/2022, 4:30 PM
    Hi may I ask how to deploy Prefect Server myself? aka
    prefect orion start
    , mainly that this web UI doesn’t require login or other form of authentication. I can’t just deploy and allow anyone on the internet can visit that.
    m
    • 2
    • 5
  • y

    yu zeng

    06/26/2022, 12:21 PM
    @Kevin Kho hi kevin . is there any bug on my prefect 1.0? the task has beed marked as successed, but later, it start again and i don't know what happen. could you help me
    k
    • 2
    • 1
  • j

    Jean-Paul Berthelot

    06/26/2022, 1:04 PM
    Hi all I would like to understand performance tuning options for
    RayTaskRunner
    versus
    DaskTaskRunner
    , I have a requirement to develop a Prefect Workflow for a high volume ingestion workflow. The architecture I am looking at is based on Data Mesh. I have a local environment and worked through several POCs similar to the workflow described here. Could I please be directed to relevent reading materials or examples?
    k
    • 2
    • 2
  • d

    Dekel R

    06/26/2022, 2:07 PM
    Hey all, I have a general question about the development cycle using Prefect in an organization. For example, we have 3 environments in our company, dev stage and prod. Dev is a classic development environment - most of the R&D team has access to it. Staging is pre-prod - when a project moves into staging it’s actually under “code freeze” - in this ENV we will also do some QA. Now the idea is to auto-register Prefect flows to dev by using CI-CD - the registering will happen after a PR gets approved (git PR), After the merge, and in order to move to staging - we plan to use tagging. After testing in staging - we plan to have a flow for moving a staging version to prod. What’s the best way to tackle this? Specifically working with Prefect (Prefect is our pipeline scheduler). Also, should we have 3 different Prefect servers (we think we do - since we want to eliminate any possibility in which dev flows will somehow affect prod ones)? 3 different cloud accounts? (we now use a single account and use Projects for different company projects and not environments) Hope my question is clear. Thanks!! @Ofek Katriel
    o
    k
    • 3
    • 11
  • j

    James Phoenix

    06/26/2022, 4:09 PM
    Heyas everyone 🙂
  • j

    James Phoenix

    06/26/2022, 4:09 PM
    https://orion-docs.prefect.io/api-ref/prefect/cli/storage/#prefect.cli.storage.create
  • j

    James Phoenix

    06/26/2022, 4:10 PM
    Looks like when I run the CLI command for this on orion, I am not able to provide a service account credential for GCP buckets.
    k
    m
    • 3
    • 7
  • j

    James Phoenix

    06/26/2022, 4:10 PM
    How can I set a service account for prefect 2.0 then?
  • j

    James Phoenix

    06/26/2022, 4:37 PM
    Perhaps the CLI for the GCP cloud storage on 2.0 needs updating to include service account credentials when creating it?
  • i

    ibrahem

    06/26/2022, 6:37 PM
    HI .. Can someone explains how to change the default configuration files locations such as storage directory and flows locations, the default is HOME/.prefect/ would like to move it to /data/prefect_server/. Can someone point me to the proper docs
    k
    • 2
    • 12
  • a

    Alfred Martinez

    06/26/2022, 8:47 PM
    Hi everyone, I am wondering if any of you know if there is a way within the prefect UI to send an email alert to people if /when a flow fails to run properly. thank you
    k
    • 2
    • 3
  • j

    Jovan Sakovic

    06/26/2022, 9:53 PM
    Hi all 👋 Just a quick question, haven’t explored much yet - curious whether it would be possible to sync the Calendar from Prefect Server with GCal 🤔 I assume this data would be available in one of the Postgres tables when tasks are scheduled?
    k
    • 2
    • 1
  • b

    Ben Muller

    06/27/2022, 3:03 AM
    Hey - I am registering a flow like with these arguments/parameters => 🧵
    k
    • 2
    • 6
  • s

    Surya

    06/27/2022, 9:07 AM
    Hi All, Incase if anyone is using the dask gateway in combination with prefect. Any idea when is the end of support date for dask-gateway 0.9.0 version
    k
    • 2
    • 1
  • m

    Marius Haberstock

    06/27/2022, 9:23 AM
    Hi, are there any SSO Options for Prefect Cloud?
    j
    c
    • 3
    • 2
  • m

    Michal Zawadzki

    06/27/2022, 9:30 AM
    Hi, is there any more detailed information available on running long-running flows in Prefect 2.0? Or are they exactly the same as regular ones and I can simply do
    while True:
    if x:
    my_task()
    And deploy & manage as usual?
    j
    k
    • 3
    • 2
  • z

    Zheng Cheng

    06/27/2022, 10:10 AM
    Small typo in Blog: https://www.prefect.io/guide/blog/getting-started-prefect-2/
    c
    k
    p
    • 4
    • 9
  • r

    redsquare

    06/27/2022, 11:33 AM
    Should prefect clean up the jobs created in k8's - created a scheduled job but the jobs get left in an 'Active' state even thought he pod is completed
    r
    k
    • 3
    • 13
  • w

    wonsun

    06/27/2022, 12:06 PM
    Hi community~! I have a question about PermissionError. I am trying to create a virtual environment in a separately created account in Linux and run prefect flow there. In my flow, there is a task that needs to go to a specific folder and save the task's result as a csv file, and a permissionError occurs in this process. Maybe I think it's normal. If i want to create, modify, or delete data in that folder, i must enter 'sudo' and something, but flow does not include that task(sudo ~~). How can i solve this problem?
    j
    k
    • 3
    • 3
  • g

    Guoying Qi

    06/27/2022, 12:12 PM
    Can you delete a storage?
    prefect storage delete my-s3-storage
    k
    • 2
    • 3
  • n

    Naga Sravika Bodapati

    06/27/2022, 12:38 PM
    seeing this error: raise exceptions.from_http_status(response.status_code, message, response=response) google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/uat-pvr-data-prefect-gcs/o?uploadType=resumable&amp;upload_id=ADPycdtTsfUe4rNUbqxBn59GhS-wvYByENDQ_Sx0uT8u1ekPASue0EnpTYTZws2ULMcX3_rkBzdvnxf9ySecwbAhZzOK205Y4YOT: Invalid request. There were 22287 byte(s) in the request body. There should have been 22839 byte(s) (starting at offset 0 and ending at offset 22838) according to the Content-Range header.: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PERMANENT_REDIRECT: 308>) when trying to use script as storage with GCS
    k
    • 2
    • 5
  • n

    Naga Sravika Bodapati

    06/27/2022, 12:39 PM
    any ideas?
  • n

    Naga Sravika Bodapati

    06/27/2022, 12:39 PM
    have both key and local storage path
  • n

    Naga Sravika Bodapati

    06/27/2022, 12:39 PM
    local script path
  • g

    Guoying Qi

    06/27/2022, 1:09 PM
    Can you use self-managed S3 server as prefect storage?
    Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
    k
    • 2
    • 3
  • z

    Zheng Cheng

    06/27/2022, 1:33 PM
    Hi, There are should be doc and example about how to trigger a flow run on Prefect Cloud with HTTP Request. This is very important to me. Not just a https://orion-docs.prefect.io/api-ref/rest-api/#prefect-rest-api-reference
    k
    • 2
    • 5
  • z

    Zheng Cheng

    06/27/2022, 1:39 PM
    The HTTP API doesn’t seem working with Prefect Cloud 2.0 (https://beta.prefect.io/api/hello) it work fine on Local Prefect Server (http://localhost:4200/api/hello)
  • z

    Zheng Cheng

    06/27/2022, 1:42 PM
    OK I get it
  • z

    Zheng Cheng

    06/27/2022, 1:43 PM
    k
    • 2
    • 5
  • z

    Zheng Cheng

    06/27/2022, 2:18 PM
    Seem like the Prefect 2.0 REST API is mostly for Orion Client? I have no idea how to use them. Seem like I could use AWS API Gateway + Lambda(Python) + Prefect to make a API endpoint, then I can send a http request to that API endpoint -> AWS API Gateway -> Lambda -> Prefect. Seem a bit over complicated
    k
    • 2
    • 4
Powered by Linen
Title
z

Zheng Cheng

06/27/2022, 2:18 PM
Seem like the Prefect 2.0 REST API is mostly for Orion Client? I have no idea how to use them. Seem like I could use AWS API Gateway + Lambda(Python) + Prefect to make a API endpoint, then I can send a http request to that API endpoint -> AWS API Gateway -> Lambda -> Prefect. Seem a bit over complicated
I think for now AWS API Gateway + AWS Lambda + Prefect to make a API endpoint instead of using Prefect Cloud Deployment would be a better choice. (easier) I just need to set Environment Variable in Lambda with PREFECT_API_URL=http://beta.prefect.io/api and PREFECT_API_KEY=xxx
I don’t need to config work queue or agent, Lambda would run Python + Prefect for me. Prefect became more of a retry + logging tools
k

Kevin Kho

06/27/2022, 3:32 PM
It’s mostly through Orion Client because there is a lot of work to shape the requests that happens so doing a manual request would repeat a lot of the work
z

Zheng Cheng

06/27/2022, 3:37 PM
understood, Thanks @Kevin Kho!
View count: 5