• Darragh

    Darragh

    2 years ago
    Has anyone managed to get Prefect installed into a Docker container for the purpose of building Flows into Docker? I keep running around in circles with the D-in-D problem, my brain is creaking…
    Darragh
    Scott Zelenka
    +1
    69 replies
    Copy to Clipboard
  • j

    John Ramirez

    2 years ago
    hey everyone - weird question but does anyone have experience using Apache Spark. I’m investigating for a project best practices to run multiple parameter models on a single data set within a Spark cluster orchestrated with prefect. My main question is where to place the multiplier; would I get better performance to submit multiple job using
    .map()
    or submit a single job and manage running the different models within the single spark job.
    j
    1 replies
    Copy to Clipboard
  • e

    Emmanuel Klinger

    2 years ago
    Hi. I'm wondering what is the recommended way to run only parts of a flow. For example only tasks with certains tags.
    e
    Zachary Hughes
    +1
    6 replies
    Copy to Clipboard
  • k

    Kaz

    2 years ago
    Hey all, has anyone experienced issues where every so often the flow doesn’t execute because it fails to load custom modules? I’m using a local agent and have added the correct import paths. My flow was working just fine for the first few runs. Now, it runs maybe 1/3 times because it fails to load one of my custom modules. This could be a supervisord issue as well, but I’ve been doing some digging around and I’m unable to pinpoint where things are going wrong. any and all help is appreciated!
    k
    1 replies
    Copy to Clipboard
  • a

    Andy Waugh

    2 years ago
    Hello 👋 can anyone help clarify my understanding how LOOPing and Context works between workers, especially (if it makes a difference) when using a distributed Dask cluster. Presumably the new task that is created is not necessarily executed on the same worker? And if not, can I safely rely on the newly LOOP’d task having the latest context? Here I’m specifically interested to ensure task_loop_result will reliably have the latest value but I am also generally interested to better understand how Context etc. is managed between workers. Hope that makes sense - feel free to point me at anything if this already explained elsewhere! Thanks! Andy
    a
    Chris White
    3 replies
    Copy to Clipboard
  • Jeremiah

    Jeremiah

    2 years ago
    If you’re curious about Vue or front-end development, @nicholas is going to give @Laura Lorenz (she/her) an introduction to Vue in this Friday’s live stream — their goal will be to add a new tile the open-source UI from scratch, resulting in a new PR for Prefect Server! All experience levels welcome, you can sign up here: https://www.meetup.com/Prefect-Community/events/270547519
    Jeremiah
    m
    2 replies
    Copy to Clipboard
  • m

    matta

    2 years ago
    What's the best way to pass an object containing credentials? I'm making an ETL to get stuff out of Google Sheets, and I'm using the
    gspread
    package, which has you do everything from method calls to an authentication object. So like you go
    gc = gspread.service_account(filename=<filename>)
    and point it at a special credentials file, then everything is through that. Should I just pass it to Secrets? Is there any risk of sensitive credentials being cached somewhere if I define the
    gc
    object within the Flow itself?
    m
    nicholas
    +2
    37 replies
    Copy to Clipboard
  • Darragh

    Darragh

    2 years ago
    Has anyone come across a case where you build a flow locally, and the register step is configured to register to a remote server (private prefect instance on aws, not prefect cloud). The output give me a url where the flow should be registered on that server, but there's nothing in the UI..
    Darragh
    Dylan
    +2
    92 replies
    Copy to Clipboard
  • s

    Simon Basin

    2 years ago
    Hello! - Task question: is it possible to 1. re-run a task w/o triggering downstream dependencies? 2. do the same with ad-hoc task parameters?
    s
    Laura Lorenz (she/her)
    5 replies
    Copy to Clipboard
  • Christopher Harris

    Christopher Harris

    2 years ago
    Does prefect support micro-batching? In more detail: We’re trying to migrate our existing generator pattern to prefect - and we’re hoping we can change to a micro-batching model. Basically, our first node in our pipeline is responsible for pulling data from a location and pushing it out to the rest of the pipeline (a DAG). We were hoping to use the LOOP construct to have that “source node” pull data in
    batch_size
    increments, and map the individual data packets across the remaining DAG. In a way this kind of seems like a “workflow loop” with the parameters for the first node constantly updating.
    Christopher Harris
    Dylan
    9 replies
    Copy to Clipboard