• Irakli Gugushvili

    Irakli Gugushvili

    11 months ago
    Hello Everyone, I wanted to ask if there is a mechanism for Batch Processing in Prefect. I have found Child/Parent flows to simulate that behavior but interested if there are some other approaches.
    Irakli Gugushvili
    Kevin Kho
    4 replies
    Copy to Clipboard
  • warmwaffles

    warmwaffles

    11 months ago
    For k8s agents, do I need to use a service account for them?
    warmwaffles
    Michael Adkins
    +1
    13 replies
    Copy to Clipboard
  • warmwaffles

    warmwaffles

    11 months ago
    For keeping my staging and production environments separate, is it wiser to create a tenant per environment or is splitting them under projects recommended?
    warmwaffles
    Kevin Kho
    5 replies
    Copy to Clipboard
  • warmwaffles

    warmwaffles

    11 months ago
    Anyone here use prefect tasks with your django environment? Just wondering how that is going to play nice since there is some song and dance you have to do to get scripts to work with django
    warmwaffles
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Aaron Ash

    Aaron Ash

    11 months ago
    Is it possible to set priorities for flows with the kubernetes agent? Ideally I'd like to have high priority flows be able to pre-empt and pause lower priority flows if the cluster is maxed out
    Aaron Ash
    Kevin Kho
    2 replies
    Copy to Clipboard
  • w

    William Grim

    11 months ago
    Hi all! We have a prefect agent that is scheduling flows when we tell it to schedule them, but then they just sit in a "Scheduled" state when we run a command like
    prefect get flow-runs -f myflow
    . We also see a few runs with
    Success
    ,
    Failed
    , and
    Cancelled
    as well. I don't know how someone got a flow to cancel though.
    w
    Kevin Kho
    8 replies
    Copy to Clipboard
  • Abhishek

    Abhishek

    11 months ago
    Hello folks, i am getting an error during a flow run for one of the long running tasks. the error is:
    No heartbeat detected from the remote task; marking the run as failed.
    Other details: • Its running on ECS (agent is ECS) • Flow storage is S3 • In the same Flow other long running tasks did run successfully What can i do to prevent this timeouts? and how to fix it. As this doc suggests Lazarus will restart the failed task. but i need a way to prevent the timeout if possible.Screenhot:
    Abhishek
    2 replies
    Copy to Clipboard
  • Guy Propper

    Guy Propper

    11 months ago
    Hi, I have a question about running two mapped tasks one after the other. I have the following flow:
    file_paths_to_process_list = extract_files_to_process()
    mapped_parsed_output = transform.map(file_paths_to_process_list)
    load_output_to_db.map(mapped_parsed_output)
    I want map#2 (load_output_to_db) to depend on results from map#1 (transform), but not to wait for all results from map#1. As soon as there is one result from map#1, it should be processed in map#2. What is the correct way to do this? Thanks!
    Guy Propper
    Henning Holgersen
    +1
    4 replies
    Copy to Clipboard
  • Henning Holgersen

    Henning Holgersen

    11 months ago
    Is there a way to do retries on entire/scheduled flows ? Retries on tasks work well, but I have had a flow fail because the executor is a remote cluster - so it doesn’t even reach the Task stage. In my case, a coiled cluster failed to spin up for some reason (actually first time that happened).
    Henning Holgersen
    Kevin Kho
    4 replies
    Copy to Clipboard
  • a

    Amar Verma

    11 months ago
    Hi, I am currently exploring options to automate ML model experimentation. Can Prefect be consider as an alternative to MLFlow for the same. Does its capabilities covers that of an MLFlow?. Thanks!!!
    a
    Kevin Kho
    2 replies
    Copy to Clipboard