• Samuel Hinton

    Samuel Hinton

    11 months ago
    Hey all! Was wondering if someone could help me out in configuring a prefect agent to make it a bit more robust. After we had an outage this evening, a massive backlog of flows occurred. This would be fine, except that the agent we have (a local agent run as a service in the docker compose) immedietly tries to launch all the delayed jobs, which makes it run out of memory. We get the Lazarus errors, the agent restarts, and same thing happens. You can see in the screenshot of service memory the constant agents restarting, using all available memory, and eventually crashing. Are there any settings/env variables I can set to try and a) limit the number of concurrent jobs the agent sends out and b) stop whatever is using memory? What exactly would cause an almost instant multiple GB memory usage in the agent?
    Samuel Hinton
    Kevin Kho
    +2
    43 replies
    Copy to Clipboard
  • Xinyi Guo

    Xinyi Guo

    11 months ago
    Hey all! I am new to prefect, recently want to run a scheduled prefect job, but seems its being killed from time to time by a prefect ZombieKiller task, with an error No heartbeat detected from the remote task; marking the run as failed. Any idea why this happens, and how to make sure my task always have a heartbeat could be detected by the prefect ZombieKiller? thank you.
    Xinyi Guo
    Anna Geller
    3 replies
    Copy to Clipboard
  • a

    Arun Giridharan

    11 months ago
    Question: Is it possible for 2 parent flows to access the same child run id?
    a
    Kevin Kho
    7 replies
    Copy to Clipboard
  • s

    serhan

    11 months ago
    Hey all! I am working on to integrate splunk with prefect agent. I need to know where all the LOGS being written, what is the path for the log files? Can anyone help?
    s
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Eric Feldman

    Eric Feldman

    11 months ago
    Hey again 🙂 I’m trying to get the params that flow was run with in a state handler method but
    new_state.result
    keeps being an empty dict when running on prefect agent if i’m running the flow using
    flow.run
    I do see the results of the state
    Eric Feldman
    Anna Geller
    3 replies
    Copy to Clipboard
  • j

    Jason Boorn

    11 months ago
    Quick question - if I have a set of tasks that don't depend on each other but that I want to run serially, what's the correct way to do that?
    j
    Anna Geller
    4 replies
    Copy to Clipboard
  • Donny Flynn

    Donny Flynn

    11 months ago
    Hi! I want to automate a simple python process with Prefect, but I have a laptop I close to take in/out of the office. What would be the easiest online agent to run a scheduled flow? Would a GKE job to poll Prefect cloud as an agent be the best/easiest way? I was exploring Coiled, but I don't think you can set that as an agent, unless I'm missing something...
    Donny Flynn
    Anna Geller
    19 replies
    Copy to Clipboard
  • Imran Qureshi

    Imran Qureshi

    11 months ago
    We're trying out the prefect cloud. How do folks handle pipelines in different environments (dev, staging, production)? do folks use teams, tenants, project etc for this?
    Imran Qureshi
    Anna Geller
    +1
    6 replies
    Copy to Clipboard
  • Vamsi Reddy

    Vamsi Reddy

    11 months ago
    Hi everyone, I am working on a flow where i need to spin up an emr cluster -> submit a step(using shell task) -> check the status of the step and read log files(using boto3)-> submit further steps based on the status/ fail the flow/step if error occurs in logs. Is there a way possible where i can change the state of a previous task to failed based on the next task?
    Vamsi Reddy
    Kevin Kho
    8 replies
    Copy to Clipboard
  • Kevin Mullins

    Kevin Mullins

    11 months ago
    I’m somewhat new to prefect. I’ve done simple mapped tasks but am curious if I can chain subsequent tasks to mapped tasks to perform a fan-out/fan-in type approach. Here is a diagram trying to visualize what I’d like to accomplish. Any guidance if this is possible and/or approaches would be greatly appreciated:
    Kevin Mullins
    Kevin Kho
    12 replies
    Copy to Clipboard