Christoph Deil
01/04/2022, 9:15 PMflow.run()
with the default serial executor happens or where the algorithm is that linearises the task graph to decide who runs after who. And also how schedulers work, i.e. if they have at the very core some polling for loop in-process or if the scheduler runs in some other thread or process. For now I’m not planning to look into Dask, just trying to understand how the serial execution works under the hood.
Does this information exist in some tutorial form? Or alternatively could you please point me to the few relevant parts in the code or tests to quickly understand how that works?Kevin Kho
Kevin Kho
Kevin Kho
Christoph Deil
01/04/2022, 9:23 PMKevin Kho
Kevin Kho
here▾
Christoph Deil
01/04/2022, 9:27 PMKevin Kho
Kevin Kho
Gus Cavanaugh
01/04/2022, 9:50 PMdavzucky
01/05/2022, 12:26 AM@task
• Developer focus on business
• Simple to scale out with Dask
• Getting inside from Prefect Server/Cloud
• Take care of the result persistanceChristoph Deil
01/05/2022, 2:42 PMwith Flow
and flow.run()
execution, how the DAG is created and executed. So my target audience was just the curious Python dev that likes to fully understand the tools he uses.
I know the strength of Prefect is that you don’t have to understand what’s happening under the hood, you can just learn the nice API to make tasks and flows. But for that part there’s already https://docs.prefect.io/ and other great content, I don’t think it would be useful if I write a similar tutorial on that.
So I gave the tutorial today for my colleagues who were mostly seasoned Python developers but Prefect newbies like myself, and it went well, we learned a bunch of stuff together, even if the part to understand what happens under the hood is still mostly to be learned at a later time. We’ll sleep on it, but mostly opinion was that we prefer Prefect Orion and for our limited use of existing flows (just running on a schedule, no server or cloud) we’d try already migrating to it, even if it’s alpha. 🙂Anna Geller
Christoph Deil
01/05/2022, 4:04 PMAnna Geller
Anna Geller
from datetime import timedelta
import pendulum
from prefect import task, Flow
from prefect.schedules import IntervalSchedule
@task
def say_hello():
print("Hello, world!")
schedule = IntervalSchedule(
start_date=pendulum.now(tz="Europe/Berlin"), interval=timedelta(minutes=1),
)
for sched in schedule.next(20):
print(sched)