@Marvin in my prefect flow, I want a task that will hit a long-running async external rest API, then poll for completion of that task by periodically hitting a status endpoint, and run another task upon completion. I do NOT want to just loop and sleep, because it may take a long time and I want to free up executors for other work. So is there some way to do this?

Oh, interesting issue you've got there. With my near-infinite capacity for boredom, waiting for long-running tasks by polling intermittently sounds like a task right up my alley. But, I understand why you wouldn't want to use your valuable executors for such a mundane task. What you're asking for can actually be achieved quite elegantly in Prefect by means of a combination of async calls and Prefect's scheduling capabilities. Here's a step-by-step approach that might work for you: 1. Asynchronous Task Execution: Create an asynchronous task to hit the external REST API. With Prefect 2.x, you can create an async function and use the

@task

decorator to turn it into a Prefect task. 2. Polling with Scheduled Runs: Instead of using sleep, schedule another task that will poll the status endpoint. This task needs to be invoked periodically based on your expected waiting time for the API task completion. 3. Conditional Execution: Use a condition to check the status from the status endpoint. On completion, trigger the next task. 4. State Dependence: Make sure your next task depends on the successful state of the polling task, ensuring the next task runs only after the long-running API task is complete. Prefect 2.0 introduces a new concept called 'blocking tasks', which can be used to wait for the completion of an external event. However, since Prefect 2.0 and its detailed workings are out of my knowledge scope and we wouldn't want to guess and cause your flow to block indefinitely, I'd suggest looking up the official Prefect documentation or checking out GitHub issues to see any associated patterns or examples for accomplishing this without a constant loop. Shall I go ahead and look for some specific examples that could guide you further?

yes

Hey @Scott Stafford.... thanks for asking this question. I'm trying to do the same thing. Did you have any luck identifying a solution?