https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • s

    Stephen Lloyd

    05/20/2022, 11:26 AM
    Hi. What is the overlap and important differences in features between Prefect Server and Prefect Cloud. I couldn’t find anything in the Prefect docs site that dealt with this. Context: We’re on Prefect Cloud now. We’re looking to upgrade our plan. Currently on a legacy plan WELL within the free tier, but we wanted to add additional users which requires setting up billing. Our procurement team got pricing that is a massive jump from $0/month. So much of a jump that we will have to either move to Prefect Server or move to AWS Managed Airflow. So trying to figure out what a change to Prefect Server might mean.
    ✅ 1
    a
    5 replies · 2 participants
  • l

    Lana Dann

    05/20/2022, 9:27 PM
    is there a way to skip a directory when registering flows with the prefect cli?
    k
    7 replies · 2 participants
  • k

    Kayvan Shah

    05/21/2022, 3:27 PM
    Is there a way to delete storage
    ../..> prefect storage
    Usage: prefect storage [OPTIONS] COMMAND [ARGS]...
    
      Commands for managing storage settings.
    
    Options:
      --help  Show this message and exit.
    
    Commands:
      create         Create a new storage configuration.
      ls             View configured storage options.
      reset-default  Reset the default storage option.
      set-default    Change the default storage option.
    a
    3 replies · 2 participants
  • b

    Benson Mwangi

    05/22/2022, 2:31 AM
    Hello, I have a prefect flow that I'm interacting with via the graphql API and it works perfectly. Now, I was wondering if it's possible to prefix the flow ID (i.e.when deploying the flow pre-fix the ID). I know this is counter intuative as this would mean there's no versioning of flows (i.e. perhaps the flow version is always the same). However, the reason I'm looking for this option is because my flows are executing tasks triggered through some APPS and I would not want to change the flow ID in each app whenever a flow is resubmitted. In short, is there a way to keep the flow ID constant or somewhat pre-fix it? Also, just for context I'm running a self-hosted prefect server. Looking forward and many thanks in advance.
    k
    1 reply · 2 participants
  • c

    Côme Arvis

    05/23/2022, 9:29 AM
    Hello! All our flows are failing for about 20 minutes, with always the same issue (screenshots in the thread) We didn’t have this error before, do you know if there is any issue in your end? Thanks!
    a
    3 replies · 2 participants
  • m

    Matt Drago

    05/23/2022, 11:15 AM
    Hey Folks, Been a while since I've frequented these slack channels, but I've finally gotten some time at work to upgrade the Prefect installation that I set up a while ago (version 0.14.14). I was looking at using the Helm charts so I could dispose of the ones that I hand crafted. One of the problems that I am trying to work out though is how to connect to the external DB. My deployment environment is a K8S cluster in GCP. In my current installation, I have configured cloud_sql_proxy sidecars for the graphql and hasura deployments. I cannot find a way to do this with the new charts. Is the only way to connect to the CloudSQL Postgres DB to use the private/public IP of the DB instance?
    a
    t
    14 replies · 3 participants
  • l

    Lana Dann

    05/23/2022, 6:02 PM
    is there a way to codify the readme for a flow? https://docs.prefect.io/orchestration/ui/flow.html#read-me
    k
    2 replies · 2 participants
  • k

    Kayvan Shah

    05/24/2022, 10:17 AM
    Is there a workaround to bind prefect logger with logger implemented in module?
    ✅ 1
    a
    t
    8 replies · 3 participants
  • m

    Michael Smith

    05/24/2022, 12:13 PM
    Hello, with Prefect 1 on K8s what are the options for partitioning workloads so that we can prioritise "interactively" triggered workflows vs pure batch?
    a
    6 replies · 2 participants
  • p

    Pedro Martins

    05/24/2022, 11:02 PM
    Hello! I’m trying to connect the Prefect Server running on k8s to a managed PostgreSQL in the Azure Cloud. The DB name, host and secret are provided through a key vault within a K8s secretproviderclass. I want now to make Hasura mounting the connection string and load the secrets from there. Can someone help me here?
    m
    2 replies · 2 participants
  • c

    Chris Hatton

    05/26/2022, 5:26 PM
    Hey all ... I'm getting an error saying
    prefect.exceptions.ClientError: [{'path': ['create_project'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    when I try to REcreate a project with the same name in
    backend cloud
    .... Is this a known issue? Does anyone know of a workaround? (Details in thread)
    k
    13 replies · 2 participants
  • j

    Javier Ebrero

    05/27/2022, 11:54 AM
    Hello all! I am trying to start the prefect server in my local using the backend as server and running prefect server start. I never had a problem using it but currently I buy other laptop and I am getting an error from hasura service (which a connection refuse to the postgres database) the traceback error is inside the thread. The prefect version is 0.15.13 and the OS version is ubuntu 22.04
    k
    a
    24 replies · 3 participants
  • s

    Shaoyi Zhang

    05/31/2022, 11:01 PM
    Hey all! What will be the road map for Prefect Server look like with the Prefect 2.0 migration? Is Prefect Server compatible with Prefect 2.0? If not, will it be compatible? or should we consider it as a dead project?
    k
    2 replies · 2 participants
  • z

    Zhibin Dai

    06/01/2022, 4:13 PM
    Hi team, we are facing an odd issue with scheduling flows. We have a python script that schedules flows. When the script gets run, it duplicates the schedule for one particular flow, for the next day. For example, if there is a flow scheduled for 1 am tomorrow, when we re-run the build_flows script, it will create another flow for that same time the next day, but not any subsequent days
    k
    8 replies · 2 participants
  • m

    Maria

    06/02/2022, 8:49 PM
    Hi, i’m having issues trying to use LocalDaskExecutor(scheduler=“threads”) with map(). When I try to call in flow,
    foo.map(list_custom_class)
    , i get an error like the following:
    Unexpected error: AttributeError("'ThreadPoolExecutor' object has no attribute 'apply_async'")
    Traceback (most recent call last):
      File "/miniconda3/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/miniconda3/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 543, in get_flow_run_state
        {e: state for e, state in upstream_states.items()}
      File "/miniconda3/lib/python3.7/site-packages/prefect/executors/dask.py", line 711, in wait
        futures, scheduler=self.scheduler, pool=self._pool, optimize_graph=False
      File "/miniconda3/lib/python3.7/site-packages/dask/base.py", line 447, in compute
        results = schedule(dsk, keys, **kwargs)
      File "/miniconda3/lib/python3.7/site-packages/dask/threaded.py", line 77, in get
        pool.apply_async,
    AttributeError: 'ThreadPoolExecutor' object has no attribute 'apply_async'
    What could be causing this error?
    k
    4 replies · 2 participants
  • d

    Denys Volokh

    06/05/2022, 3:25 AM
    Hi, we have Prefect Server running in kubernetes. I can open UI but not able to create a project. Here is API response in the browser:
    a
    1 reply · 2 participants
  • l

    Lana Dann

    06/06/2022, 7:25 PM
    is it possible to have a flow that has a dynamic number of tasks? for example: task 1: get a list of values for each value, kick off a task
    k
    m
    7 replies · 3 participants
  • d

    Diego Alonso Roque Montoya

    06/06/2022, 11:55 PM
    Is there a known reason why kicking/waiting for sub flows in prefect 1 fails in the main flow even though the kicked flows actually succeed?
    k
    7 replies · 2 participants
  • d

    Diego Alonso Roque Montoya

    06/06/2022, 11:57 PM
    We have gotten a bunch of those with the new task type (more specifically no heartbeat detected errors) and it makes our data flow very brittle
  • s

    Saurabh Indoria

    06/07/2022, 5:24 AM
    Hi all, We are facing this issue on Prefect randomly..
    Pod prefect-job-04b1fb0c-chzc9 failed.
    	Container 'flow' state: terminated
    		Exit Code:: 137
    		Reason: OOMKilled
    Can some elaborate as to what this error is about and how can we rectify this? Our Stack: Prefect Cloud + K8s Agent + LocalDaskExecutor
    ✅ 1
    a
    2 replies · 2 participants
  • a

    ash

    06/07/2022, 1:34 PM
    Hello everyone , I have my prefect server running on K8 with a k8 agent. I want to integrate a dask execution layer for the already deployed system. Can anyone guide me with a simple way to do that. Presently i register my flows from local machine and i am using git storage for storing scripts.
    a
    1 reply · 2 participants
  • j

    James Keegan

    06/07/2022, 1:40 PM
    Hey! We're getting a lot of these errors on our K8s agent and it's preventing the server and agent communicating as expected. Any quick fixes for this? Error in thread 🧵
    ✅ 1
    a
    8 replies · 2 participants
  • d

    Denys Volokh

    06/07/2022, 3:09 PM
    Hi, I am not able to add secrets from UI and from Interactive API, what could be wrong here?
    k
    1 reply · 2 participants
  • d

    Denys Volokh

    06/07/2022, 3:35 PM
    How can I unlock all menu items in the Team menu?
    k
    3 replies · 2 participants
  • p

    Prasanth Kothuri

    06/08/2022, 9:21 AM
    How can I deploy a flow from my mac/desktop to remote prefect server ? currently I am running
    python3 flow_name.py
    on the server where prefect is installed, thanks !
    ✅ 1
    a
    8 replies · 2 participants
  • b

    Binoy Shah

    06/08/2022, 1:28 PM
    is the Orion Server Helm chart published to Bitnami ? https://github.com/PrefectHQ/prefect-recipes/tree/main/helm/orion-server
    ✅ 1
    a
    g
    5 replies · 3 participants
  • l

    Lana Dann

    06/08/2022, 10:56 PM
    is it possible to patch a task of a flow when testing a flow run using pytest?
    ✅ 1
    k
    1 reply · 2 participants
  • c

    cramos

    06/09/2022, 2:59 PM
    Orion with Cloud 2.0: Hello, i have a deployment set up with a cron scheduler (runs every 5 minutes). I noticed that if i shut down the agent, lets say for an hour or so, and start it up again, all the scheduled runs during that hour will be launched. Is possible to limit this behavior? Like if scheduled run is like X minutes late, mark it as a failure?
    ✅ 1
    k
    3 replies · 2 participants
  • m

    Maria

    06/09/2022, 4:40 PM
    Hi, where can I configure Lazarus behaviour? We are getting this error
    A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
    while we are waiting for nodes to free up
    k
    22 replies · 2 participants
  • c

    Christian Nuss

    06/09/2022, 7:53 PM
    Hey there! Quick question! We're doing a ShellTask with • return_all=False, • stream_output=True, • log_stderr=True, inside the ShellTask, the command is a pipe operation,
    command1 | command2
    .
    command1
    exits with a non-zero code, how do i get the ShellTask to also fail?
    ✅ 1
    k
    2 replies · 2 participants
Powered by Linen
Title
c

Christian Nuss

06/09/2022, 7:53 PM
Hey there! Quick question! We're doing a ShellTask with • return_all=False, • stream_output=True, • log_stderr=True, inside the ShellTask, the command is a pipe operation,
command1 | command2
.
command1
exits with a non-zero code, how do i get the ShellTask to also fail?
✅ 1
k

Kevin Kho

06/09/2022, 8:04 PM
I think this is specific to shell behavior like this ?
c

Christian Nuss

06/09/2022, 8:31 PM
ah pipefail yes, thank you!
View count: 5