https://prefect.io logo
k

Kevin Weiler

01/04/2023, 10:22 PM
Hi there! I’m looking at transitioning to Prefect 2.0 and trying to wrap my head around some of the new stuff. In my existing workflow, I construct a flow programmatically by walking a set of yaml files. This has the effect of needing the
add_task
method from the
flow
class. I don’t think the new API has this. Is there a way to iteratively add tasks to a flow?
I suppose I could do some trickery with closures or something
m

Mason Menges

01/04/2023, 11:05 PM
Hey Kevin could you provide some more context for what you're trying to achieve by "adding" tasks to the flow, generally speaking if you just need to loop through files and perform the same or similar actions on them you should just be able to write a for loop to do so from a single task
k

Kevin Weiler

01/05/2023, 2:29 AM
hi @Mason Menges - sure! We allow users in our organization to add entries to a set of yaml files in a directory. Each entry defines a job with the job name, resource requirements (cpu/mem), the docker image they should run out of, an entrypoint, and crucially any other jobs it might depend on upstream. Each job launches a batch run in our docker orchestration tool (
nomad
- it’s a lot like
kubernetes
) and polls the api until the job either times out, fails, or completes successfully. I load these yaml files into a single dictionary, loop through them once to add them to a flow, and then loop through them again to set dependencies. What I’m left with is a single flow with a few different subtrees. I then register the flow with the prefect api and run them on a schedule.
We could, of course, ask our users to define a python function for their job instead of an entry in a yaml file. But these users are researchers, and if you give them python - they’ll use (and abuse) it. We use this yaml file approach to keep rather strict guard rails up around what they can and cannot do.
my interest in Prefect 2.0 is splitting out the subtrees into subflows to allow for greater operational flexibility through the gui (maybe, still evaluating)