• a

    Andrew Schechtman-Rook

    2 years ago
    Hi folks! I've been poking around a bunch with Prefect for the last couple months, and over that time I've written a few utilities/extensions to help Prefect work better for my particular data and modeling workflows. In case anyone else may be able to benefit I've pulled them together into a small package — https://pypi.org/project/prefect-ds/. Please feel free to take a look 🙂
    a
    Jeremiah
    +1
    19 replies
    Copy to Clipboard
  • j

    John Ramirez

    2 years ago
    if i want to improve performance through server selection, is memory or compute optimized better for dask?
    j
    Jeremiah
    2 replies
    Copy to Clipboard
  • n

    Nico Aiello

    2 years ago
    Hey everyone, I'm new to the prefect community and was hoping to get advice on how to use prefect for my particular use case. Suppose we first have some batcher task that takes a set of items and batches them into groups. We then need a task to run on each batch. (So far, this can be handled using prefect's mapping). Next, we then want to run a task for each item in the batch and we'd like to be able to run this as soon as each batch finishes, rather than waiting for all batches to finish. To visualize a simpler version:
  • n

    Nico Aiello

    2 years ago
    . --------> item a1 ------------> batch a --------> item a2 batcher --------> item a3 ------------> batch b --------> item b1 --------> item b2 Basically, I'd like prefect to be aware that once the batching happens, we now have two independent, parallelizable flows happening - so if batch a finishes, item a1 can begin even if batch b is still in progress.
    n
    Jeremiah
    +1
    11 replies
    Copy to Clipboard
  • j

    John Ramirez

    2 years ago
    Hey everyone, is there a task setting to prevent a mapping task from running until all upstream tasks are complete?
    j
    Dylan
    18 replies
    Copy to Clipboard
  • m

    Mark Williams

    2 years ago
    Im trying to determine the best route for reading in a list of text files using Prefect. Would I use the LOOP signal to loop over the list of files or should I build my flow as pertained to one file that way my flow represents a specific file that was processed?
    m
    i
    8 replies
    Copy to Clipboard
  • a

    Andor Tóth

    2 years ago
    hi everyone
    a
    i
    5 replies
    Copy to Clipboard
  • a

    Andor Tóth

    2 years ago
    I was wondering, after thoroughly looking in the documentation, that how could I execute a single task on it’s own from a flow?
    a
    Dylan
    +1
    4 replies
    Copy to Clipboard
  • a

    Andor Tóth

    2 years ago
    Do you have any suggestions?
  • Braun Reyes

    Braun Reyes

    2 years ago
    context option on flow run in cloud is 🔥 could totally use this to pass things to remote environment and agent for running tasks from cloud!
    Braun Reyes
    Arsenii
    2 replies
    Copy to Clipboard