• Gagan Singh Saluja

    Gagan Singh Saluja

    9 months ago
    Hi , if the server goes down for some reason, how to ensure that the data is persistent? Because every time I close and start the server I loose the track of the previous runs and metrics in UI
    Gagan Singh Saluja
    Amanda Wee
    +1
    14 replies
    Copy to Clipboard
  • jack

    jack

    9 months ago
    How to track reruns? We've been calling
    client.create_flow_run()
    to create several flow runs (using ECSRun), and then polling each with
    client.get_flow_run_state
    to know when all the flows have completed. When one of the flows fails (and prefect starts a new flow run to take its place), how can we check when the rerun is complete (and whether it succeeded)?
    jack
    Kevin Kho
    17 replies
    Copy to Clipboard
  • jack

    jack

    9 months ago
    How to detect state == FAILED as soon as heartbeat fails? We have a flow set up to run on ECS that purposefully consumes all the available memory on the container. (We want to make sure we can handle such edge cases.) We monitor the prefect logs, and this message comes through:
    No heartbeat detected from the remote task; marking the run as failed.
    For 20+ minutes following that log message, fetching the flow run state from prefect cloud still shows
    <Running: "Running flow.">
    Ideally, as soon as the flow run is marked as failed, state from prefect cloud would say Failed. Suggestions?
    jack
    Kevin Kho
    +1
    25 replies
    Copy to Clipboard
  • l

    Lukas N.

    9 months ago
    Does
    CloudFlowRunner
    support graceful shutdown? We're running flows as Kubernetes jobs on EC2 spot instances, which get terminated from time to time. Let's say the job starts a pod
    a
    on a node that terminates. Kubernetes will quickly reschedule it and spawn a pod
    b
    , but
    b
    does nothing because the state of the tasks is
    Running
    , but they are not (
    a
    is dead). We need to wait for the heartbeat to timeout for Prefect to reschedule it which takes a long time. Instead setting something like
    Pending
    or even
    Failed
    to tasks that are
    Running
    when a SIGTERM is received would be nice.
    l
    Kevin Kho
    +1
    9 replies
    Copy to Clipboard
  • Benson Mwangi

    Benson Mwangi

    9 months ago
    Hi, I'm new to prefect and I was wondering if it's possible to trigger a flow through the graphql endpoint. I am currently self-hosting a prefect server and would like to trigger a flow from a separate system running off a UI. Would appreciate any insights or materials that can lead me to a solution. I've successfully installed prefect and able to execute some flows (via linux terminal) including scheduled flows but what I'm looking for is the ability to trigger a flow externally via the graphql api. Sorry for the beginner question 🙂
    Benson Mwangi
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Sylvain Hazard

    Sylvain Hazard

    9 months ago
    Hey everyone ! Hope you had a great weekend ! I'm having troubles regarding flows memory footprint. I have a bunch of flows that look like this :
    ids = get_ids()
    step_1 = do_step_1.map(ids)
    step_2 = do_step_2.map(step_1)
    step_3 = do_step_3.map(step_2)
    The number of ids retrieved can vary by a few orders of magnitude and I cannot predict it. The issue I see is that while the flow runs, the memory footprint keeps increasing which sometimes results in an OOM kill of the pod running the flow. Is there any way to have the memory footprint be near constant with regard to the number of executed mapped tasks in the flow ? I understand that the initial mapping requires a bunch of memory and that there is no way around it. I am running on K8S, using a LocalDaskExecutor (threaded) and had hoped that depth first execution would mean there would be some amount of garbage collecting with fully executed branches. Would setting up a
    Result
    in the mapped tasks help in any way ? I tried manually releasing memory inside the tasks code (with
    del
    and
    gc
    mostly) but saw no improvement. Another solution I see would be to have steps 1-3 be executed in their own separate flow but that means we would spin up a bunch of ephemeral pods and lengthen the flow overall I suppose ? Thanks !
    Sylvain Hazard
    Anna Geller
    +1
    3 replies
    Copy to Clipboard
  • r

    Robert Hales

    9 months ago
    Hi there, had a few tasks finish with the state on
    Pending
    which I think lead to double runs. From what I understand
    Pending
    shouldn’t be a terminal state?
    r
    Anna Geller
    37 replies
    Copy to Clipboard
  • Guilherme Petris

    Guilherme Petris

    9 months ago
    Hey! I’m trying to figure it out why my flows are registering the run in less than a second, but in this situation if a run the script separately they run for a couple of minutes.
    from analytics_toolbox.analytics_toolbox import *
    from analytics_toolbox.analytics_toolbox import PendoUtils as pendo_utils
    from analytics_toolbox.analytics_toolbox import SnowflakeUtils as snowflake_utils
    from prefect import task, Flow
    from prefect.utilities.notifications import slack_notifier
    
    @task ()
    def main():
    ......
    with Flow("pendo_featureEvents") as flow:
        main()
    
    flow.register(project_name="pendo_api")
    Guilherme Petris
    Anna Geller
    9 replies
    Copy to Clipboard
  • f

    Faizan Qazi

    9 months ago
    Hi guys. I am running prefect server locally and trying to access it using the ip_address which i also configured in the
    config.toml
    file . But as i access it at
    <http://ip.add.re.ss:8080>
    , the page displays but then redirects to the getting started page . So basically i want to access my prefect instance from any other machine.
    [server]
      [server.ui]
      apollo_url="<http://ip.add.re.ss:4200/graphql>"
    f
    Anna Geller
    15 replies
    Copy to Clipboard