https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    Margaret Walter

    10/21/2021, 5:36 PM
    Hey! I was wondering if anyone had modified the helm chart to use Ingress instead of LoadBalancers for the GraphQL and Apollo endpoints and whether you ran into any issues with it.
    k
    m
    • 3
    • 2
  • j

    Jacob Bedard

    10/21/2021, 6:02 PM
    Hi everyone! I'm a noob to Prefect, working with cloud, and I'm wondering how to grab one of my secrets saved in my prefect cloud team's secrets while I'm testing locally
  • j

    Jacob Bedard

    10/21/2021, 6:03 PM
    I haven't found a code example that actually reaches into a cloud team and collects a secret. It looks like "use_local_secrets" needs to be passed to an argument, but I'm not sure where. For now, I just want to set this stuff for my current python session.
    k
    • 2
    • 14
  • j

    Jacob Bedard

    10/21/2021, 6:03 PM
    Can anyone help me out?
  • w

    William Jamir

    10/21/2021, 6:51 PM
    Hi, does someone knows or recommend any material/course/book that uses prefect?
    k
    • 2
    • 1
  • a

    Aqib Fayyaz

    10/21/2021, 9:01 PM
    prefect agent kubernetes install -k API_KEY | kubectl apply --namespace=my-namespace -f -
    this command successfully creates the deployment and referrs to default prefect image so the question is how can i use my code with it?
    k
    a
    • 3
    • 83
  • t

    Tony Yun

    10/21/2021, 9:04 PM
    hi, does anyone know how to get the flow running status inside a task? I need make some logics for different status like cancelled, running, etc.
    m
    • 2
    • 29
  • h

    haf

    10/21/2021, 9:58 PM
    How can I handle "internal error" in Prefect cloud?
    Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
        task_run_info = self.client.get_task_run_info(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1798, in get_task_run_info
        result = self.graphql(mutation)  # type: Any
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
        raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]
    m
    z
    +3
    • 6
    • 124
  • w

    William Grim

    10/21/2021, 10:36 PM
    Potentially related to https://prefect-community.slack.com/archives/CL09KU1K7/p1634766491212000 on a long-running flow, we are no longer able to receive the logs for that flow. Any time I use
    prefect get logs --id uuid_of_that_flow
    , it gets killed with no logs retrieved. Is there another place to go look for logs?
    m
    • 2
    • 10
  • t

    Tony Yun

    10/21/2021, 10:46 PM
    I don’t know if this is a bug or not - I set a task concurrency to 1 with one label defined and testing, however, it keeps hanging at state:
    Queued
    with just one that task running. Only if I increase the limit to 2 it will start running. It looks a bug to me:
    k
    a
    • 3
    • 40
  • m

    Maria

    10/21/2021, 11:05 PM
    Hi! I'm trying to figure out what is the easiest way to restart the flow that crashed (but hasn't started)? For a context, we are facing concurrency issue
    An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
    and I'm aware there is open github issue (https://github.com/PrefectHQ/prefect/issues/4402). We did adjust params as recommended but it still happens. I want to be able to restart the flow but cannot use task retry feature as no tasks have been started (it is the only log line I see and there is no start time for the flow ):
    k
    • 2
    • 6
  • a

    Adam Everington

    10/22/2021, 7:57 AM
    Hey guys, i'm running prefect server on an azure vm. 2 issues, first is how I set the --expose flag in the docker-compose.yaml and i'm also struggling to connect to the ui remotely. The ports have been exposed and ip's have been whitelisted to our internal org only
    a
    • 2
    • 21
  • e

    Eddie Atkinson

    10/22/2021, 9:12 AM
    Hi folks, I am looking at interfacing with prefect’s graphql API using Typescript and was wondering if the schema for Prefect’s API is available anywhere? Ideally I’d like to autogenerate typescript types for the queries, mutations etc. I’ve tried using
    graphql-code-generator
    but that hasn’t worked so far
    a
    • 2
    • 7
  • h

    haf

    10/22/2021, 10:50 AM
    What's the recommended way to add monitoring and observability to a prefect flow? I mean: • the ability to see input-output for tasks (at least for a while after running them; just not Results) • the ability to see timings for each task • any structured logging from inside tasks (e.g. especially important segment variables / which app / tenant is it running for) • integrating with Prometheus ?
    a
    • 2
    • 2
  • a

    Adrien B

    10/22/2021, 12:32 PM
    Hi everyone, I am trying to define a new flow. In this flow I have a task that I map on thousands of values. The problem is that some children tasks fail because there are too many open connections simultaneously. When I run my flow on 1000 arguments it works fine but on 8000, some child tasks are stuck in a running state. Is there a way to do the mapping consecutively by chunks of 1000 arguments ? Basically, If I give a parameter list of 5000 arguments to my flow, I would like the flow to run the task five times consecutively. Each time the task would me mapped over 1000 arguments. I don’t have to use the results of the tasks if that would be an issue. The flow would run on a localdaskexecutor with 15 proceseses and use a prefect server as an endpoint/host. Thank you so much for your help guys
    a
    k
    • 3
    • 6
  • t

    Toprak Nihat Deniz Öztürk

    10/22/2021, 1:50 PM
    Can I access results of a flow run created with create_flow_run. I normally start child runs with
    f_run = flow.run
    method and access the results with
    f_run.results[df].result
    .
    k
    • 2
    • 3
  • t

    Thomas Furmston

    10/22/2021, 3:04 PM
    Hi. Sorry to ask such a basic question, but is it possible to delete a flow from the prefect server? I would like to do it programmatically, but I can't find any reference to it the documentation. I also can't see this option in the prefect cloud webUI.
    k
    • 2
    • 8
  • j

    João Luiz Carabetta

    10/22/2021, 3:57 PM
    Hi, I was just trying the prefect orion tutorial (https://orion-docs.prefect.io/tutorials/first-steps/) but it failed 😕 It failed on this line
    state = my_favorite_function()
    k
    • 2
    • 3
  • s

    Sahil Chopra

    10/22/2021, 5:15 PM
    Hi everyone, when creating an
    EmailTask
    what is the
    EMAIL_USERNAME
    and
    EMAIL_PASSWORD
    used for? Are the notifications shipped from this email? Is this required, or is it possible to use the default notifications@prefect.io email address shown in the class definition?
    k
    • 2
    • 4
  • h

    Hugo Kitano

    10/22/2021, 5:28 PM
    Hi, I’m trying to ping the graphql API with python and javascript and running into some parsing problems. How would I convert the following to python using the requests package?
    query {
      flow_run(where: {name: {_eq: "woodoo-leopard"}}) {
        id
        state
        start_time
      }
    }
    k
    • 2
    • 4
  • e

    Erik Nilsson

    10/22/2021, 5:55 PM
    Hi, Is there an equivalent of Airflow’s `TaskGroup`’s in Prefect? Or is there any other way to build larger flows (100+ tasks) while making it easy to navigate the flow in the UI? I think I’ve seen something similar before, but I can’t find any references… Thanks
    k
    • 2
    • 2
  • g

    Giovanni Giacco

    10/22/2021, 6:03 PM
    Hi, I created a flow of flows. Is there a way to call subflows with StartFlowRun depending on values from a input Parameter. For example input parameter has values [a, b, c] so I need to call StartFlowRun for flow “a”, “b” and “c”
    k
    • 2
    • 2
  • c

    Charles Liu

    10/22/2021, 6:29 PM
    What are our options for external monitoring or webhook integrations for total system uptime for Prefect? Looking for the ability to alert for outages.
    k
    • 2
    • 8
  • j

    Jacob Goldberg

    10/22/2021, 6:53 PM
    I have many tasks distributed across files, and one slack_notifier handler that they all import and utilize. What is a good pattern for disabling/using a different handler for local dev. I cannot run flows locally unless i change the handler because i get a
    ValueError: Local Secret "SLACK_WEBHOOK_URL" was not found.
    k
    • 2
    • 9
  • k

    Kevin Kho

    10/22/2021, 7:56 PM
    Hey everyone, join us for a fireside chat with our CEO Jeremiah and CTO Chris about Orion. They will be answering community questions. Direct link

    here▾

    ✅ 1
    :prefect: 1
    👍 1
  • t

    Tony Yun

    10/22/2021, 8:01 PM
    hi, is there a way to kill task in flow context? I know how to start a task in there, fairly easy. But not to kill a started one.
    k
    • 2
    • 9
  • d

    Dylan

    10/22/2021, 8:06 PM
    LiveStream starts now! 😛arty-parrot:

    https://youtu.be/etktOF6Wb4o▾

  • a

    Abhas P

    10/22/2021, 8:09 PM
    Hi Team, Is there a way in prefect.core library/api to de-register or delete stale flows?
    k
    • 2
    • 6
  • j

    Joe Still

    10/22/2021, 9:04 PM
    Howdy Prefect team, I found you via Open MLOps and keep seeing you pop up in community discussion. Is there somewhere I can reference for how Prefect executes my code? Like a sequence diagram or something to help me wrap my head around it? Enjoyed the fireside, Orion sounds awesome!
    👋 2
    k
    • 2
    • 2
  • m

    Matt Alhonte

    10/22/2021, 11:17 PM
    So I'm refactoring a Flow into a Flow of Flows. It runs a bunch of Papermill notebooks. They have to be run in a certain order - but in some cases, a dependency isn't relevant, so we skip it. This currently takes the form of a bunch of Tasks along the lines of
    if notebook_name in list_of_notebooks:
       run_notebook(notebook_name)
    else:
        print("skipped")
    How do I do that with the
    StartFlowRun
    task? Should I subclass it and replace the existing
    .run()
    method?
    k
    • 2
    • 3
Powered by Linen
Title
m

Matt Alhonte

10/22/2021, 11:17 PM
So I'm refactoring a Flow into a Flow of Flows. It runs a bunch of Papermill notebooks. They have to be run in a certain order - but in some cases, a dependency isn't relevant, so we skip it. This currently takes the form of a bunch of Tasks along the lines of
if notebook_name in list_of_notebooks:
   run_notebook(notebook_name)
else:
    print("skipped")
How do I do that with the
StartFlowRun
task? Should I subclass it and replace the existing
.run()
method?
k

Kevin Kho

10/22/2021, 11:26 PM
Subclassing the run makes sense here. Then you can just trigger the
StartFlowRun
call if it fulfills the condition.
m

Matt Alhonte

10/22/2021, 11:29 PM
Cool, thanks! So I guess I'll move the current
.run()
fn to a different name (let's say
.execute_flow
, and then have the new
.run()
as something like
if notebook_name in list_of_notebooks: self.execute_flow()
?
k

Kevin Kho

10/23/2021, 1:48 AM
You can override the run and use super right? The DBTShellTask is a good example of this as it just overrides the ShellTask then calls
super.run()
to call the ShellTask
run()
👍 1
View count: 2