https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • a

    Axel Guinot

    07/01/2022, 5:07 PM
    Hello 👋, I am running Prefect-1 on a distant cluster. My Flows are working well and I can monitor them from the Prefect Cloud. Though, I would like to try doing it using a Prefect server. The issue is that I do not have docker on my cluster (and I cannot install it) so I cannot start a server there. I was wondering if it would be possible to run a prefect server on my local machine and connect to a running agent on my cluster through ssh tunnel or similar? If anyone has ever done it I would really appreciate some pointers or an example on how to setup something like that. Thanks 🙂
    ✅ 1
    a
    • 2
    • 3
  • v

    Vipul

    07/02/2022, 5:02 PM
    Hi, I was just going through the Prefect Orion release notes 2.0b7, it says Ray and Dask are now part of collection - “_Split Ray and Dask task runners into their own collections”_ Though if I look at the requirements.txt, I could see the reference of the Ray package - The reason I am raising is because I am using slightly older version of Red Hat that does not work with the Ray and so I had to pip install prefect without dependency and if the Ray is removed from the requirements, that would make my life easy, thanks. https://github.com/PrefectHQ/prefect/blob/orion/requirements.txt#L29
    ✅ 1
    a
    • 2
    • 3
  • a

    ash

    07/05/2022, 4:45 PM
    Hello Everyone, I am running prefect server on K8s. I am checkpointing certain tasks but not all. Seems like when the flow breaks and i restart it, the flow is running from the very beginning rather than the point where code broke. How can make sure the flow restarts from the task where it failed #prefect-server
    ✅ 1
    k
    • 2
    • 32
  • t

    Tomás Emilio Silva Ebensperger

    07/06/2022, 8:48 PM
    I am running prefect 2.0 on aws lambda (runtime python3.9) but i get an error
    "Orion requires sqlite >= 3.24.0 but we found version 3.7.17"
    Does anyone know a workaround? i
    ✅ 1
    k
    m
    a
    • 4
    • 12
  • s

    Slackbot

    07/07/2022, 7:16 AM
    This message was deleted.
  • e

    Elliot Oram

    07/07/2022, 8:25 AM
    Hey all, I'm getting an error upgrading from prefect
    1.2.2
    to
    1.2.3
    Installation appears to work as expected but I get the following when I run
    prefect server start --postgres-port 5433
    Has anyone else encountered this or did I miss some migration notes somewhere?
    Untitled.cpp
    ✅ 1
    a
    • 2
    • 2
  • a

    Abin Joseph

    07/08/2022, 7:34 AM
    Hi all, Using prefect version: 1.2.3 Whenever I set the
    performance_report_path
    argument in
    DaskExecutor
    it fails with the following error:
    Unexpected error: ValueError('No global client found and no address provided')
    prefect report.txt
    a
    k
    • 3
    • 7
  • b

    Benson Mwangi

    07/08/2022, 9:32 PM
    Hi, Using prefect 1.X and able to trigger a flow via a GraphQl endpoint (see sample response below). I was wondering if it's possible for a prefect flow to return a response (e.g. computed values or a generated file). { "data": { "create_flow_run": { "id": "b7f44e94-d899-4f1b-a854-98dff845a0e2" } } } Thank you
    k
    • 2
    • 3
  • a

    Architha Rao

    07/11/2022, 6:05 AM
    Hey all. I have a flow that is scheduled to run every day but I keep getting this error which I don't comprehend where to trace it to: [{'message': "can't start new thread", 'locations': [{'line': 2, 'column': 5}], 'path': ['set_flow_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "can't start new thread"}}}] Any idea why? I'm using Prefect Core Version: 0.13.18
    ✅ 1
    a
    • 2
    • 1
  • a

    Abin Joseph

    07/11/2022, 11:30 AM
    Hi all, is it possible to have different executors per step? This is useful when resource requirements are different for steps
    ✅ 1
    a
    • 2
    • 5
  • o

    Octopus

    07/13/2022, 8:54 AM
    [v1.2.1] Hello, where can I find the IP configuration for containers ? Edit: I may be the docker proxy that I need to configure to do this, no necessarily via Prefect containers , idk When I do
    docker port tmp_ui_1 8080/tcp
    I get
    127.0.0.1:8080
    but I would like to change it to
    0.0.0.0:8080
    :::8080
    ✅ 1
    a
    • 2
    • 7
  • e

    Eric So

    07/14/2022, 7:57 AM
    Is it possible to control how Prefect traverse the DAG? e.g. depth first? An example use case would be, I would like to prioritise all child mapped tasks for a give parent to complete first, e.g. if I had 10 tasks each map out to 10 child tasks, and I had 10workers, I would like it to complete one parent task and all 10 child mapped tasks at a time, before moving onto next, rather than start a child of each parent task, if that make sense?
    ✅ 1
    a
    • 2
    • 2
  • v

    Vaibhav Shetye

    07/14/2022, 8:46 AM
    How do I update the prefect server url in the Slack Notification ? At present it points to http://localhost:8080/default/flow-run/687d73b7-112b-4fca-b8c1-73c7fb12e6ae, how do I update the server ip address ? is there some config parameter ?
    ✅ 1
    a
    • 2
    • 2
  • c

    Christian Nuss

    07/14/2022, 5:50 PM
    hey all! question! we have 3 flows kick off at the same time, all 3 flows are slightly different and take different times to execute. once all 3 flows are finished, we want to kick off ONE flow that would process data that the other 3 produced. ideally that one flow would be allowed to run once the other 3 flows are finished
    a
    • 2
    • 5
  • b

    Bernardo Galvao

    07/15/2022, 9:03 AM
    hey, I triggered these ad-hoc by clicking the play button runs and they are in scheduled state since forever.
    m
    • 2
    • 5
  • s

    Sergey Dashko

    07/15/2022, 1:09 PM
    Hello. Is it possible to use
    S3List
    task to get list of bucket specifying full path? Whenever I try to provide full path there is an error about bucket not matching some Regex. It works only when I provide plain name of the root bucket like this
    test-bucket
    but I need content of the directory inside that bucket. How this can be done?
    a
    • 2
    • 2
  • m

    Marcus Hughes

    07/16/2022, 4:19 AM
    We're looking to transition from Prefect 1.0 to 2.0 when it's generally available. We run on our own servers instead of using Prefect Cloud. Despite being adorable, we opted to not use the default flow run names (e.g. delightful-marten, portable-scallop, strong-muskrat) and have our flows rename themselves with names that describe properties of their data. In Prefect 1.0, we used the
    RenameFlowRun
    task. How do we do the same thing in Prefect 2.0? I don't see a similar task but maybe I just missed it. Thanks for the help!
    a
    • 2
    • 1
  • a

    Anders Segerberg

    07/18/2022, 4:46 PM
    We are on
    1.1.0
    . I cancelled a running flow through the UI; it was deep in the middle of running a series of mapped tasks. Turns out it continued to run for 6 days before ending with
    [11 July 2022 2:49pm]: No heartbeat detected from the flow run; marking the run as failed.
    What is the expected behavior when "cancelling" a flow engaged in executing mapped tasks? (We are also on the dask-executor, with thread-based parallelism.)
    k
    • 2
    • 11
  • l

    Lucien Fregosi

    07/19/2022, 9:29 AM
    Hi I have a question regarding prefect running on kubernetes through helm template deployment. I have troubleshooting connecting the prefect UI with the apollo server. I use a clusterIp service to be able to communicate between the pods. I’m running a service on top of apollo with cluster IP
    10.1.70.59
    . When I exec the ui pod I can succesfully reach
    curl 10.1.70.59:4200/graphql
    However within the Prefect UI I can’t connect to the URL. Any idea why ?
    m
    • 2
    • 11
  • k

    kevin

    07/19/2022, 2:24 PM
    Hi guys, does Prefect's
    case
    statement for True/False logic allow me to do something analogos to this python logic:
    a = get_val_or_none()
    if not a:
        a = 'some new val'
    do_something_with_a(a)
    k
    • 2
    • 3
  • a

    Alexander van Eck

    07/19/2022, 3:38 PM
    Question on dynamically creating flows; Say I have 3 tasks
    @task
    def task_1():
      print('1')
    
    @task
    def task_2():
      print('2')
    
    @task
    def task_3():
      print('3')
    And I wanted to give the user the ability to say in which order these should be executed, or who should consume which one. i.e. I want to user to describe a graph of these callables and then for the flow to execute them. How would I do this?
    k
    • 2
    • 3
  • s

    Sandeep Dass

    07/19/2022, 5:51 PM
    Question about prefect server graphql endpoint. Getting the below error. Any ideas as to why
    k
    n
    a
    • 4
    • 13
  • s

    Sam Pibworth

    07/20/2022, 2:17 PM
    Hello, got a problem with Prefect server UI not being able to connect to core. The toml file has the line
    apollo_url = "http:/<ip of azure vm>:4200/graphql"
    . Any help would be much apperciated!
    k
    • 2
    • 6
  • a

    Abin Antony

    07/20/2022, 6:52 PM
    Hello, I am setting up PREFECT on my Openshift environment refering PREFECT Helm (https://github.com/PrefectHQ/server/tree/master/helm/prefect-server#installing-released-versions). BY default, I can see prefect backend is set as CLOUD. My question is , Any options available to set up it with defaulted as "SERVER"
    a
    • 2
    • 1
  • j

    Jacob Longhurst

    07/21/2022, 7:33 PM
    Can anyone help me work out a bug? I’m building the docker image with prefecthq/prefect:1.1.0-python3.8. I’m also registering the flow with prefect 1.1.0 and doing so using the cli. The error is
    Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("Can\'t get attribute \'_make_function\' on <module \'cloudpickle.cloudpickle\' from \'/usr/local/lib/python3.8/site-packages/cloudpickle/cloudpickle.py\'>")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n  - cloudpickle: (flow built with \'2.1.0\', currently running with \'2.0.0\')')
    k
    • 2
    • 6
  • k

    Kelvin

    07/25/2022, 11:28 AM
    Hi! Just a question about DeploymentSpec, been searching through the docs and slack and still a bit confused. I'm on orion beta 2.0b6 about to move up to latest: • What will happen to my existing old deployments, will I need to redeploy them? Thanks!
    m
    • 2
    • 1
  • l

    Lana Dann

    07/25/2022, 5:26 PM
    does anyone know the best way to look up how a flow was kicked off? (manually, via schedule, via run flow task) i assume we’d be able to find this info using the graphql api
    m
    • 2
    • 2
  • k

    Karanveer Mohan

    07/26/2022, 9:04 PM
    Hi all, Just updated to 2.0b14 and ran into a couple of issues: 1/ Running
    prefect cloud login -k KEY -w WORKSPACE
    now asks for
    "Creating a profile for this Prefect Cloud login. Please specify a profile name:"
    This breaks my deployment flow since I need it to be programmatic. I also tried running
    prefect profile create
    before but that doesn't seem to help. I also can't give the name of an existing profile. Could someone help explain what I'm doing wrong? I tried searching in the docs but maybe I'm missing something 2/ I was able to just hit enter locally and got the following message "Logged in to Prefect Cloud using profile ''." However, if I try to do
    prefect cloud login
    or
    prefect profile inspect
    after this, it throws the following error. I had to fix this by manually deleting
    profiles.toml
    . Code in thread
    a
    d
    b
    • 4
    • 8
  • j

    Jacob Longhurst

    07/27/2022, 3:02 AM
    I’m hoping someone can help me with getting Dask Distributed working inside a k8s cluster. We have our flows configured to use a KubernetesRun. The run config and the executor are using the same image with a base image of
    prefecthq/prefect:1.1.0-python3.8
    . If we use the
    DaskExecutor
    with a custom class and kwargs, we get an error message saying we need to install kubectl. If we choose to use a
    LocalDaskExecutor
    it works just fine. It seems it’s failing some where around attempting to talk between the dask scheduler and workers. We’ve created the roles and role bindings dask requires in order to function properly and we’re not sure if we need to pass in a dask scheduler spec and / or service spec to the dask KubeCluster object? Hoping someone has been able to solve running a temporary, on demand distributed dask cluster in K8S with the KubernetesRun prefect run config
    a
    • 2
    • 1
  • a

    Anna Geller

    07/27/2022, 2:18 PM
    archived the channel
Powered by Linen
Title
a

Anna Geller

07/27/2022, 2:18 PM
archived the channel
View count: 9