https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • v

    Vadym Dytyniak

    07/25/2022, 12:31 PM
    Hi. We use KubernetesRun and see debug logs like this in Prefect Cloud Logs:
    Event: 'FailedScheduling' on pod 'prefect-job-79c82cd5-w6nfd'
    	Message: 0/7 nodes are available: 2 node(s) had taint {<http://dask.corp.com/component|dask.corp.com/component>: scheduler}, that the pod didn't tolerate, 5 node(s) didn't match Pod's node affinity/selector
    Do you have ideas how to disable them?
    r
    • 2
    • 9
  • s

    Steph Clacksman

    07/25/2022, 1:20 PM
    I can't seem to get tasks running concurrently or in parallel with dask in prefect 2 without using task.submit() Is that expected behaviour? It doesn't seem to be reflected in the docs.
    ✅ 1
    t
    s
    • 3
    • 3
  • t

    Tom Klein

    07/25/2022, 1:28 PM
    Hello 🙋 is there a default limit for concurrent flows or something in the Cloud? we have currently a flow that was run ad-hoc and is still running, and a flow that was scheduled to run now, and it’s stuck in
    Submitted
    and nothing seems to be happening with it
    m
    • 2
    • 6
  • d

    David

    07/25/2022, 1:47 PM
    Hi all, I am trying to run a flow with VertexRun. I created a GitLab repo, configured the Gitlab storage. Run a agent locally with vertex and the prefect cloud api key. However on run I keep getting the same error. Failed to load and execute flow run: ModuleNotFoundError. It keeps refering to a local path.
    ✅ 1
    b
    • 2
    • 1
  • d

    David

    07/25/2022, 1:47 PM
    Why do I keep seeing this error while I am using the gitlab storage...
  • r

    Rajvir Jhawar

    07/25/2022, 2:01 PM
    Hi prefect team, I had a question about deployment pausing in prefect 2.0. I am little confused by this concept. If a work queue is filtering by deployment and the deployment is paused do all flow runs based on said deployment get paused and go into a "pending" state? I don't see a way "pause" deployment programatically via either the cli or the api. It seems like this is a UI only feature am i missing something here?
    ✅ 1
    j
    • 2
    • 2
  • p

    Prass

    07/25/2022, 2:13 PM
    Shameless plug: I made a meta-package linking different packages for different use cases. https://github.com/prassanna-ravishankar/python-productivity-stack Under the
    mlops
    section, I've added prefect -
    pip install productivity-stack[mlops]
    ⭐ 3
    🚀 3
    a
    j
    • 3
    • 2
  • c

    Chu

    07/25/2022, 2:54 PM
    Hi, anyone knows how can I send params which contain mapped and unmapped parameters to create_flow_run? (More in the thread)
    ✅ 2
    a
    m
    • 3
    • 33
  • a

    Andrew Pruchinski

    07/25/2022, 3:34 PM
    Hey all. Our flow isn't able to spin up more than 180 jobs. We can't determine the root cause exactly but at a certain point there's a connection aborted and prefect starts to remove jobs and then marks the entire flow as failed. We are using a daskexecutor on the cloud. Looking into the dask repos, we aren't able to set some debug level logging as an attempt to retrieve some info on what’s really causing the
    Connection aborted
    issue. Even if there are some failures, we do want the flow to continue and the task triggers are set accordingly. Thank you
    ✅ 1
    b
    • 2
    • 19
  • l

    Leon Kozlowski

    07/25/2022, 4:13 PM
    Hi all, does anyone have experience integrating prefect alerts/failures with opsgenie?
    👀 1
    ✅ 1
    m
    • 2
    • 1
  • m

    Mikkel Duif

    07/25/2022, 4:48 PM
    Hi is it possible to clear all data from Prefect (2.0b12) on the hosted version? Similar to locally doing
    prefect orion database reset -y
    ?
    👀 1
    ✅ 1
    j
    r
    +2
    • 5
    • 9
  • m

    Mikkel Duif

    07/25/2022, 4:48 PM
    also, i am no longer able to see
    Deployments
    in Orion UI locally. If i inspect the network, I see it will successfully return my deployments, but they are never rendered in the UI.
    ✅ 1
    j
    • 2
    • 3
  • j

    Jenia Varavva

    07/25/2022, 5:16 PM
    Hi, I’m curious about how robust is Prefect operation w/
    Kubernetes Agent
    ,
    KubernetesRun
    and
    DaskExecutor
    in the face of various components outages. E.g. what will Prefect be able to recover on its own, and which case would require a manual intervention, if during a flow execution one of the following dies: prefect server (I suppose it’s ok as it’s stateless), kubernetes agent,
    KubernetesRun
    job executing a flow, dask scheduler, dask workers.
    ✅ 1
    m
    m
    • 3
    • 6
  • g

    Gunther Wallach

    07/25/2022, 5:47 PM
    Hi y'all I am developing a flow and have come across an error that neither me or my team understands. When trying to register the flow, we get:
    ValueError: Flow could not be deserialized successfully. Error was: ValueError('A task with the slug "Parameter" already exists in this flow.')
    This shows up after the step:
    [2022-07-25 17:27:11-0700] INFO - prefect.Docker | Pushing image to the registry..
    Parameters are currently in their own params.py file, and are referenced in the main flow file with:
    import <main_directory>.tasks.params as params
    ✅ 1
    b
    • 2
    • 2
  • m

    Mikkel Duif

    07/25/2022, 6:01 PM
    is it possible to add back the UUID on the
    prefect deployment create
    command? we use it to compare created deployments with existing deployments to clean out unused deployments.
    ✅ 1
    k
    a
    • 3
    • 3
  • t

    Tyler Wanner

    07/25/2022, 6:07 PM
    Hey y'all--happy Monday! I'm looking for guidance on writing credentials from Prefect Secrets to files, or alternatively for best practices using DBT profiles in Prefect flows in this Discourse topic, would love to hear from other Prefect community users: https://discourse.prefect.io/t/creating-volume-mounts-or-files-from-prefect-secrets/1243
    👀 1
    👋 2
    ✅ 1
    • 1
    • 1
  • d

    Divya

    07/25/2022, 6:11 PM
    Hello, After running the
    prefect orion start.
    It stops running quickly as below: Please let me know how do I troubleshoot this. Is it because of the windows machine that I am using?
    ✅ 1
    j
    • 2
    • 10
  • a

    alex

    07/25/2022, 7:32 PM
    Quick question: For prefect 1.0, should the env vars set in the agent's deployment manifests be present in executed flow's job environment? ?
    ✅ 1
    m
    • 2
    • 3
  • t

    Tom Klein

    07/25/2022, 8:51 PM
    Hello 🙋 Today i introduced our data-scientists to Prefect, and their main question was - what would be the easiest way for them to run their custom code (with all its various dependencies) from the flows they create Is containerizing their code in a docker image and pushing it to a remote repository (like ECR) the only reasonable way to achieve that? or could we conceivably just fetch the code from Github, and install the necessary dependencies as actual tasks in the flow, before executing it? (obviously it adds complexity and run-time to the flow, but are there any other reasons this might not work or be a bad idea?) (the main reason dockerizing and pushing to ECR is inconvenient for them is because they can't do it themselves, and we don't have a lot of DevOps resources that can easily provide them with a complete CI/CD pipeline for every new project they wanna test out)
    ✅ 1
    j
    t
    • 3
    • 15
  • c

    Chu

    07/25/2022, 9:08 PM
    can we use localdaskexecutor for parallel flows managed by k8s agent when moving to production?
    ✅ 1
    b
    • 2
    • 1
  • m

    Michael Reynolds

    07/25/2022, 9:40 PM
    I am having trouble with an error message I am receiving. I believe it has to do with the engine analyzing domain model classes i provide as input to a
    @flow
    . I will attach a micro example of what i am trying to accomplish... would someone mind helping me understanding the error message:
    FAILED my-prefect-project/tests/flows.py::test_pipeline - TypeError: __init__() missing 2 required positional arguments: 'host' and 'port'
    ✅ 1
    a
    a
    • 3
    • 15
  • l

    li liang

    07/26/2022, 6:38 AM
    After installed Prefect on Windows10, I got a error message when run 'prefect version'
    Version:             2.0b12
    API version:         0.7.0
    Python version:      3.9.1
    Git commit:          18d44c8c
    Built:               Fri, Jul 22, 2022 4:23 PM
    OS/Arch:             win32/AMD64
    Profile:             default
    Server type:         <client error>
    ✅ 1
    s
    a
    • 3
    • 8
  • m

    Mike Kovetsky

    07/26/2022, 8:43 AM
    Hey community! Thank you for your software 🙂 I have an issue with setting up my orion workflow on kubernetes. I am following this doc, but stuck at Configure storage part for my GCP bucket. The doc suggests to call
    prefect storage create
    in cli, but my 2.0b13 doesn’t have no such command returning the error Error: No such command ‘storage’. Should I create the storage using this doc ? There is no example for GCP, i found this one in the community. But there is no
    from prefect.blocks.storage import GoogleCloudStorageBlock
    import in my venv 😞
    🙌 1
    :thank-you: 1
    m
    r
    • 3
    • 11
  • h

    haris khan

    07/26/2022, 9:30 AM
    can one agent run multiple work-queues in prefect 2.0
    ✅ 1
    a
    a
    • 3
    • 10
  • t

    Tyndyll

    07/26/2022, 10:20 AM
    I have a task which will return a list, is it possible to use case based on the length of the returned list? e.g.
    data = returns_list()
    with case(data, len(data) > 1):
        do_stuff_with_data(data)
    
    with case(data, len(data) == 0):
        do_stuff_with_no_data()
    ✅ 1
    a
    • 2
    • 2
  • l

    Lucien Fregosi

    07/26/2022, 12:00 PM
    Hi 🙂 Does prefect 2.0 export some metrics through an open-telemetry exporter for example ? 🙂 And more globally what is the best way to be alerted when a job fails ? (I have already an alert in my kubernetes cluster if a pod fails but maybe it’s possible to have something more prefect related/high level)
    ✅ 1
    s
    g
    • 3
    • 4
  • b

    Bhupesh Kemar Singh

    07/26/2022, 12:28 PM
    Hi Community, I am facing issue while scheduling prefect flows (v 1.2.2). issue screenshot attached. looking for help. Thanks !
    👋 1
    ✅ 1
    b
    a
    g
    • 4
    • 7
  • b

    Bhupesh Kemar Singh

    07/26/2022, 12:33 PM
    ✅ 1
  • d

    Dominik Wagner

    07/26/2022, 1:28 PM
    Prefect 2.0 | Accessing current flow state Hi! 👋 Possibly simple question which I can’t seem to find an answer for: Can I access the state of the current flow from within that flow? In particular I’d like to check if the flow is retrying, and if so pass an extra argument to my task(s) i.e.
    @flow
    def my_flow():
      if <somehow_get_state_of_this_flow?> == "Retrying":
        do_something()
    ✅ 1
    a
    • 2
    • 8
  • d

    Divya

    07/26/2022, 1:41 PM
    Hello, I have a question regarding Prefect2.0. My team was using Prefect1.0 earlier and had been using the Graph QL functionality. Do we have this functionality in Prefect 2.0 as well?
    ✅ 1
    s
    t
    s
    • 4
    • 7
Powered by Linen
Title
d

Divya

07/26/2022, 1:41 PM
Hello, I have a question regarding Prefect2.0. My team was using Prefect1.0 earlier and had been using the Graph QL functionality. Do we have this functionality in Prefect 2.0 as well?
✅ 1
s

Stéphan Taljaard

07/26/2022, 1:44 PM
Hi. The GraphQL API has been swapped out for a REST API
:party-parrot: 1
:thank-you: 1
d

Divya

07/26/2022, 2:31 PM
Thank you for getting back @Stéphan Taljaard! If possible can you please provide the documentation for the same?
t

Taylor Curran

07/26/2022, 2:39 PM
Hi Divya, Check out these docks for an Overview of 2.0 APIs. Check out this if you are just interested in checking out the Prefect 2.0 REST API.
🙌 2
d

Divya

07/26/2022, 3:22 PM
Thank you sharing the documentation. Will go through!
s

Sebastian

07/27/2022, 9:38 AM
Out of curiosity, what was the motivation for abandoning GraphQL in favour of REST?
t

Taylor Curran

07/27/2022, 1:16 PM
Hi @Sebastian 🙂
Our CTO Chris White did a great job of answering this question live the other day.

Here's a link to the timestamp▾

of that recording.
In summary
• Lots of people were unfamiliar with GraphQL and asking for RestAPI
• For Prefect Server specifically, because we chose to use GraphQL, that’s the reason why Prefect Server is so heavy - we needed the query schema off of the database (which is what Hasura does)
• For our business logic, we had all of our mutations defined in Python code
• Those are two separate GraphQL schemas so we had to stitch them together
• All of this resulted in a very large, heavy container that was Prefect Server
• REST is very lightweight and allows us to automatically pull in Prefect flows into the UI even if they ran before the UI was launched
• Finally, GraphQL is very hard to manage when exposing publicly to a high volume of usage
🙏 2
s

Sebastian

07/27/2022, 1:26 PM
Thank you @Taylor Curran for the great insights 🙂
:gratitude-thank-you: 1
:hattip: 1
View count: 5