https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    Marcelo Ortega

    08/30/2022, 2:35 PM
    Hi there! We are seeing that when registering flows on Prefect1, the code inside the task gets executed. This makes some trouble for example when doing queries taking long time, as the task registering sort of times out. Is it thought to be this way?
    👀 1
    b
    12 replies · 2 participants
  • k

    karteek

    08/30/2022, 3:53 PM
    Not sure if this is the right channel to post this but prefect cloud seems to be very unresponsive/slow in the last few hours. We aren't able to deploy any new flows and having problems scheduling existing ones. Wanted to know if anyone at @Prefect has any insights regarding the problem. We have some critical jobs production jobs scheduled on prefect.
    t
    j
    +4
    12 replies · 7 participants
  • m

    Matt Melgard

    08/30/2022, 3:54 PM
    Hey all, has anyone seen this error when attempting to use a GCS storage backend?
    RuntimeError: File system created with scheme 'gcs' from base path '<gcs://mattmelgard-test-bucket-01>' could not be created. You are likely missing a Python module required to use the given storage protocol.
    This is the CLI command I’m using to build the deployment:
    prefect deployment build tutorial-flow/flow/tutorial-flow.py:my_flow -n tutorial-flow-k8s -t test -i kubernetes-job -sb gcs/tutorial-bucket
    and I have a block defined for the bucket with the name
    tutorial-bucket
    but I’m not sure what could be missing here, I have pretty much all the python packages installed via
    pip install 'prefect[all_extras]'
    ✅ 1
    k
    4 replies · 2 participants
  • a

    Anna Geller

    08/30/2022, 4:26 PM
    @Tim Enders I hope you don't mind that I move your message here - the other channel is for sharing tutorials I have this snippet from Prefect 1.0. What are the analogs to
    results
    and
    state_handler
    in 2.0?
    with Flow(
        "Subscriptions API Import",
        result=GCSResult(bucket="platformsh-vendor-prefect"),
        # executor=LocalExecutor(),
        executor=LocalDaskExecutor(scheduler="threads", num_workers=5),
        state_handlers=[flow_failure],
    ✅ 1
    t
    l
    9 replies · 3 participants
  • f

    Florent VanDeMoortele

    08/30/2022, 4:46 PM
    Hi there I'm working on migration from Prefect 0.x to 2.x. To deploy, I'm using
    prefect.environments.storage.Docker
    to create an image with my flow, context, scheduler and all the required virtual environnement. Then I register flow to Prefect with
    prefect.environments.DaskKubernetesEnvironment
    as environment. I'm using
    prefect.executors.DaskExecutor
    to run it. To deploy on 2.x, I simply try to use
    DockerContainer
    but it doesn't work without storage. So I try to install main branch of Prefect because of a merged PR ( https://discourse.prefect.io/t/how-to-build-deployments-with-flow-code-and-dependencies-being-baked-into-a-docker-image/1341 ) to use
    DockerPackager
    but it's not compatible with
    Deployment.build_from_flow
    . I also try to push directly my Docker image with
    push_image
    but push doesn't already work. So, I'm a bit confused because I can't find explicit documentation on Discourse and I read different responses because of the different releases on 2.x. Is
    DockerPackager
    the best practice to deploy a completely self contained flow? In this case, do you have any idea of the release roadmap to fix this (and to update documentation) ? If not, what can I should use? Thank you !
    a
    1 reply · 2 participants
  • d

    Dylan McReynolds

    08/30/2022, 5:25 PM
    Hi. I using prefect 2.2. I have a use case that looks something like this: • A data file has been created, I create a flow_run using the client API with the file path as a parameter • At the end of the flow, if it has not errored, then I want more tasks to run, but I need it to wait for a particular interval (day, week) I could do this with my own external scheduling system (that I have not written), but I would love to do that within prefect if possible. So my question…within a
    flow
    , is there a way to schedule a subflow to start at a particular time in the future? Or some other way to accomplish a long-delayed subflow/task?
    t
    d
    +1
    16 replies · 4 participants
  • s

    Shaoyi Zhang

    08/30/2022, 5:51 PM
    Prefect 1.0 - Is there a recommended way to reroute/duplicate the pip logs for
    EXTRA_PIP_PACKAGES
    to Cloud UI? When using Kubernetes agent, those logs only show up in job pods logs and are not available in Cloud UI.
    r
    a
    9 replies · 3 participants
  • r

    Renan Matias

    08/30/2022, 5:52 PM
    [RESOLVED] Hi Guys, sorry if my question is very basic, but I'm starting in prefect 😅. I'm trying to find out how I can send an HTTP request to the prefect API to create a flow that runs from a deployment. With Orion running locally, I was able to perform this request through the url
    <http://127.0.0.1:4200/api/deployments/{id}/create_flow_run>
    and it worked normally (I used Insomnia). Now, I need to make this same request through Prefect Cloud. I was already able to create the deployment and I was able to run the flow normally through the Prefect UI with the agent active. Can you help me?
    ✅ 1
  • a

    Ashley Felber

    08/30/2022, 6:23 PM
    Hello, i’m just setting up Prefect 2.0 and i’m trying to understand options for setting up agent/work queues in a production environment. I haven’t been able to find resources on pros/cons of different options. Any suggestions on where to look?
    👀 1
    b
    1 reply · 2 participants
  • m

    Marcos

    08/30/2022, 6:27 PM
    Hi! I’ve searched in the docs but couldn’t find an answer so I’m asking here, is there a way to run a task even if a previous task fails? Here’s an example:
    @flow
    async def test():
        connected = await connect_db.submit()
    
        task1 = await db_stuff.submit(wait_for=[connected])
        await disconnect_db.submit(wait_for=[task1])
    If
    db_stuff
    fails,
    disconnect_db
    never is executed but I’d like it to do it
    👀 1
    ✅ 1
    b
    1 reply · 2 participants
  • a

    Amol Shirke

    08/30/2022, 7:01 PM
    Hello, I am setting PG db for orion. I don't have super user rights on db. I am able to create tables but creating extensions is problem. How can I know what extensions orion will require and is there anything else while setup which require SU right and outside of normal schema objects? Thanks
    ✅ 1
    r
    5 replies · 2 participants
  • j

    Josh

    08/30/2022, 7:28 PM
    in 1.x, is it possible to get the labels of the flow run? I want to create sub-flows with the same labels, but can’t find a way to dynamically get the parent flow run’s labels to pass to the sub-flow.
    👀 1
    ✅ 1
    b
    2 replies · 2 participants
  • j

    John

    08/30/2022, 9:11 PM
    Just curious, any plans to support Webhook notifications via... MS Teams 😅?
    ✅ 1
    r
    4 replies · 2 participants
  • s

    Sophia Alice

    08/30/2022, 9:16 PM
    Hi! I have a Prefect flow that runs successfully and returns a
    Flow Run Succeeded
    but the process doesn’t exit after success and instead hangs. Any idea what’s happening?
    n
    5 replies · 2 participants
  • m

    Matt Melgard

    08/30/2022, 9:18 PM
    Hey all, so I’m using the Kubernetes Agent to run flows and I was wondering what if any support there is for defining some kind of DAG like workflow where parallel tasks can run in separate pods? Is that something that Prefect can do with the kube agent? Feel free to RTFM me if I missed something critical in the docs, just not finding anything myself
    ✅ 1
    n
    6 replies · 2 participants
  • j

    Jeffrey Lai

    08/30/2022, 11:07 PM
    Hi Community, In Prefect Cloud 1.0, is it possible to replace the current version of the flow when publishing? The reason I ask is that I made a few changes to my code that can resolve the failed task. But when republishing the revised flow, Prefect creates a new version of the flow, meaning I can't restart the failed task but reruning the whole flow. thank you.
    ✅ 1
    m
    1 reply · 2 participants
  • a

    Adam Brusselback

    08/31/2022, 12:41 AM
    Hey all, just getting back around to re-working my migration project from Prefect 1x to 2x. Hitting an issue when trying to register a SqlAlchemy Configuration Block: https://github.com/PrefectHQ/prefect-sqlalchemy/issues/28
    ✅ 1
    a
    5 replies · 2 participants
  • c

    Chris L.

    08/31/2022, 4:56 AM
    Hello Prefect community! Wondering if it's possible to have a custom flow run name for the scheduled flow? Alternatively, is there any way #prefect-ui for the UI to show which deployment the scheduled flow is associated with?
    ✅ 1
    a
    2 replies · 2 participants
  • l

    Low Kim Hoe

    08/31/2022, 5:17 AM
    Hi All, I am using prefect CLI to login the prefect cloud by using
    $ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    But i get the following error
    Unable to authenticate with Prefect Cloud. Please ensure your credentials are correct.
    My env is pipenv. And I am using prefect 2.2.0
    $ prefect --version
    2.2.0
    And I have created the API Key here
    ✅ 1
  • l

    Low Kim Hoe

    08/31/2022, 5:20 AM
    ✅ 1
    :thank-you: 1
    j
    2 replies · 2 participants
  • y

    yuvipanda

    08/31/2022, 6:29 AM
    Hey folks! I'm trying to figure out how to best deploy a Prefect Agent on our kubernetes cluster along with our JupyterHub, and can't really seem to find any resources on just deploying the agent 😞 Any pointers?
    ✅ 1
    j
    o
    2 replies · 3 participants
  • y

    yuvipanda

    08/31/2022, 6:29 AM
    Also I notice https://docs.prefect.io/tutorials/kubernetes-flow-runner/ comes up in Google Search if I try 'Kubernetes Prefect 2.0' but it's a 404
  • y

    yuvipanda

    08/31/2022, 6:31 AM
    I'm generally pretty confused - I have a k8s server, and I want to hook that up to prefect cloud. And I can't seem to find how. Would appreciate any help / pointers, particularly if I can do so via helm
    ✅ 1
  • a

    Abimbola Adebayo

    08/31/2022, 6:46 AM
    Hey all, new to prefect ^_^ I’m trying to make use of the github block - does anyone know where i can find documentation on this? (more specifically, how i can go from the block being set up in the UI to referencing this within a task)
    ✅ 1
    j
    5 replies · 2 participants
  • i

    iñigo

    08/31/2022, 7:40 AM
    Hello, I've tried the new 2.X version of prefect and it works with my scripts migrated. but as I have a lot of scripts in a machine already I'd like to do the migration the smoothest way. Could I have in the same machine prefect0.15 and 2.0 in differents virtual environments? thanks
    ✅ 1
    j
    1 reply · 2 participants
  • o

    Olivér Atanaszov

    08/31/2022, 8:29 AM
    Hi, is there a way to set a timeout for the K8s jobs executing the Flow Runs? I'm using 1.xx
    ✅ 1
    c
    1 reply · 2 participants
  • s

    Satsuki Nagae

    08/31/2022, 8:56 AM
    Hello. I’m using ECS Agent and dask executor using fargate cluster to parralize multiple ML model training. When I use dask executor, flow as ECS Task use DescribeTask API to dask scheduler so many time. CloudTrail show API call from Prefect ECS Task as 500/minute. So API call throttling error occur many times. Is there any way to reduce using DescribeTask API call when I use dask executor?
    1️⃣ 1
    c
    2 replies · 2 participants
  • r

    Rikimaru Yamaguchi

    08/31/2022, 9:43 AM
    hello. I am using sqlalcemy with prefect2.0. However, I am in trouble because the log appears twice. (from prefect, and, from sqlalcemy) Do you know any solution? thank you,
    ✅ 1
    j
    2 replies · 2 participants
  • c

    Chris L.

    08/31/2022, 9:59 AM
    Hello Prefect engineers. Experiencing what might be a regression bug from the new 2.3.0 release. This error does not occur in Prefect 2.2.0. Nothing about the flow or deployment was changed between the Prefect update. Unfortunately, I am unable to create a minimally reproducible example without revealing our company code...but happy to share it with a Prefect engineer over a call. Thanks in advance! However, been using the new arguments for deployments in our CICD pipeline and it's been working great 🙏
    :thank-you: 1
    ✅ 1
    j
    s
    5 replies · 3 participants
  • c

    Chris Keeley

    08/31/2022, 10:28 AM
    hi, we have a number of tenants on Prefect 1.0, do we have to explicitly ask for them to be enabled on Prefect 2.0? I couldn't see any docs about this, are tenants now workspaces?
    ✅ 1
    c
    4 replies · 2 participants
Powered by Linen
Title
c

Chris Keeley

08/31/2022, 10:28 AM
hi, we have a number of tenants on Prefect 1.0, do we have to explicitly ask for them to be enabled on Prefect 2.0? I couldn't see any docs about this, are tenants now workspaces?
✅ 1
c

Christopher Boyd

08/31/2022, 2:14 PM
HI Chris, there are a number of resources of available on this - some of this can flex depending on if you are in a self-hosted / self-served capacity versus an enterprise capacity - https://discourse.prefect.io/t/how-to-get-started-with-prefect-cloud-2-0/539
https://docs.prefect.io/migration-guide/#next-steps
c

Chris Keeley

08/31/2022, 2:20 PM
thanks, I had a look at that but there doesn't seem to be any occurrence of the word tenant in any of the docs, so it's hard to orient myself. i believe we're using enterprise, and the tenants were created for us. do we have to create them as workspaces now?
c

Christopher Boyd

08/31/2022, 2:23 PM
Hi Chris, if you’re using Enterprise, feel free to send an e-mail to cs@prefect.io - we have some resources available to get you situated 🙂
✅ 1
View count: 1