@Marvin in prefect 3, I have job A runs at 1am and a job B depend on job A run at 2am. I could create 2 separate scheduled flow deployments. But how to show their dependency? Or what other way to do this?

To schedule flows in Prefect 3.0 with dependencies, where Job B waits for Job A to complete before running, you have a few options: 1. Chaining Flows: Use a parent flow to orchestrate the sequence. You can design a parent flow that runs Job A first and then, upon its successful completion, runs Job B. This is achieved by creating tasks within your flow that run these sub-flows sequentially. Prefect handles dependencies at the task level, so this structure makes logical dependency explicit. 2. State Handlers: Implement state handlers to trigger Job B when Job A completes successfully. This requires a more manual setup of states and transitions. 3. Use Prefect's Schedules with State Management: This involves creating standard schedules but enhancing them with state checks to only trigger Job B if Job A was successfully completed. More advanced configuration can manage expected start times and dependencies through Prefect’s deployment settings. Here's a useful document on interacting with API and scheduling in Prefect, providing low-level details on managing flow runs programmatically that you might find helpful. Currently, direct scheduling

@Marvin in a flow, can a task hold off executing until a certain time like 9pm?

To delay the execution of a task in Prefect until a specific time, such as 9pm, you can implement a custom solution since there's no direct built-in feature for time-based delays in tasks. Here's a strategy you could employ: 1. Customized Logic in Task: You can include a delay mechanism within your task. This involves checking the current time and implementing a looping wait or using a sleep function until the desired time is reached. However, this could be inefficient in terms of resource utilization. 2. Use of Scheduling and State Management: If a task depends on running at a specific time, configure the task to check a condition before proceeding, and have the flow scheduled closer to the required time. This way, when the flow is triggered by the schedule, the task will check if it's time to execute. If not, it may re-run or wait as configured. 3. External Schedulers with Prefect APIs: Use Prefect APIs to create a flow run exactly at your desired time and structure your tasks based on this schedule. For a deeper dive on task writing in Prefect, you might find it helpful to explore this detailed Prefect documentation on writing tasks. Keep in mind that holding execution open with sleep for extended periods might be inefficient, akin to a device checking the refrigerator repeatedly to see if it's 9pm yet.

@Marvin Hi, I have • job A runs at 1am daily • job B runs at 3am Monday, Tuesday, Wednesday • Job B depends on data from Job A (A pulls in some files on the disk) Job A/B could either Flow/Task, but I think Flow is best because I could schedule it. My question is, how do I see the dependency between Job A and Job B? I cannot create a parent flow and wrap Job A & Job B together, because Job B doesn't always run when Job A runs. So, I cannot see the dependency like that. How should I approach this? Thank you!

To manage dependencies between jobs (flows) with different schedules in Prefect, you can use a more dynamic approach using custom triggers or conditional logic. 1. Custom Triggers: You can set up custom triggers in Prefect to handle dependencies. For example, Job B can be set to trigger only after Job A completes successfully, and only on the specified days. This is accomplished using Prefect’s

Event

and

Trigger

capabilities, which can be detailed further in the Prefect's documentation on custom triggers. 2. Conditional Logic in Flows: You can utilize conditional logic within Job B to check whether Job A has run and completed successfully on the required days. This is particularly useful if Job A's completion outputs a specific file or status that Job B must verify before executing. 3. Dependency Management through External Systems: If Prefect's built-in support isn't quite enough, using an external state or database check could help. For example, Job A could set a flag in a database that Job B checks as part of its condition to run. Unfortunately, Prefect's UI does not currently provide a visual representation of dependencies between independently scheduled flows unless they are nested within the same parent flow. For bugs or feature requests, such as interpreting flow dependencies or more intuitive scheduling capabilities, you might want to monitor Prefect's GitHub issues for any updates on these topics. In any case, defining dependencies between flows across different schedules can often feel like trying to fit a square peg in a round hole, but clever use of Prefect's flexible architecture can yield effective solutions.