Hello Prefect community! I'm new here, exploring Prefect's features for the first time. I have a question. Can Prefect tasks be long-running listener tasks, sort of like Sensors in Airflow? Any recommendations or best practices to accomplish this?

Hi @Martha Edwards, you are exactly right with both. Number 1 functions as an Airflow listener and you can do that. Our paradigm is more event-driven where if you have an AWS action for example, you can trigger a lambda, than then triggers an off-schedule flow run. So I guess if on the task level, use polling. If on the flow level, you can use event driven.

Thanks, wow quick response! 🙂 yes I already came across this article for event driven flows, which is very useful. We also need event-driven async tasks. Polling would work for us but being able to close a task from the API would be even better.

You might be able to with the GraphQL API by using the set_task_run_state mutation. I think that might work, but it seems hard to get the task identifiers from a remote location. I think you need a guarantee only one copy is running at a time. I think Flows dont have a server to support an API so you need to the Prefect Cloud’s API and hope it propagates down to the Flow. I have not tried this though. Polling will work for sure though and some of our tasks in the task library use polling

Ah I see, interesting, thanks! A quick follow-up question, are there any gotchas to be aware of when planning flows that will involve very long-running tasks, like >1 week?

I dont believe so but I would just encourage intermittent logs for debugging and you should also know Prefect 2.0 will have better support for long running flows. Think:

while cond:
    submit_task()

because it doesnt need a DAG

Oh cool, good to know! Anywhere I can read about Prefect 2.0 plans and timeline?

Initial blog: https://www.prefect.io/blog/announcing-prefect-orion/

Cool!! Thank you again for the resources and the quick response, this has been very informative.

Of course!

@kevin we did build out a prototype of using the GraphQL API to indicate that a task is closed. And as you predicted, the issue with this is that the Flow itself is still running the task (doesn't appear to know it's been closed), while the task appears successful from the Prefect Cloud UI.