https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Martha Edwards

03/04/2022, 9:58 PM
Hello Prefect community! I'm new here, exploring Prefect's features for the first time. I have a question. Can Prefect tasks be long-running listener tasks, sort of like Sensors in Airflow? Any recommendations or best practices to accomplish this?
As a complete newbie, my first two thoughts are: 1. In the task, we could write a while loop that polls until the condition is met 2. In the task we could write an infinite while loop, and then close the task with an API call. This is more event-driven
k

Kevin Kho

03/04/2022, 10:01 PM
Hi @Martha Edwards, you are exactly right with both. Number 1 functions as an Airflow listener and you can do that. Our paradigm is more event-driven where if you have an AWS action for example, you can trigger a lambda, than then triggers an off-schedule flow run. So I guess if on the task level, use polling. If on the flow level, you can use event driven.
m

Martha Edwards

03/04/2022, 10:04 PM
Thanks, wow quick response! 🙂 yes I already came across this article for event driven flows, which is very useful. We also need event-driven async tasks. Polling would work for us but being able to close a task from the API would be even better.
k

Kevin Kho

03/04/2022, 10:06 PM
You might be able to with the GraphQL API by using the set_task_run_state mutation. I think that might work, but it seems hard to get the task identifiers from a remote location. I think you need a guarantee only one copy is running at a time. I think Flows dont have a server to support an API so you need to the Prefect Cloud’s API and hope it propagates down to the Flow. I have not tried this though. Polling will work for sure though and some of our tasks in the task library use polling
m

Martha Edwards

03/04/2022, 10:08 PM
Ah I see, interesting, thanks! A quick follow-up question, are there any gotchas to be aware of when planning flows that will involve very long-running tasks, like >1 week?
k

Kevin Kho

03/04/2022, 10:11 PM
I dont believe so but I would just encourage intermittent logs for debugging and you should also know Prefect 2.0 will have better support for long running flows. Think:
Copy code
while cond:
    submit_task()
because it doesnt need a DAG
You can’t use the while look like that in Prefect 1 currently
m

Martha Edwards

03/04/2022, 10:15 PM
Oh cool, good to know! Anywhere I can read about Prefect 2.0 plans and timeline?
k

Kevin Kho

03/04/2022, 10:16 PM
Timeline is beta release end of this month so not production ready quite yet
m

Martha Edwards

03/04/2022, 10:18 PM
Cool!! Thank you again for the resources and the quick response, this has been very informative.
k

Kevin Kho

03/04/2022, 10:20 PM
Of course!
m

Martha Edwards

03/08/2022, 10:16 PM
@kevin we did build out a prototype of using the GraphQL API to indicate that a task is closed. And as you predicted, the issue with this is that the Flow itself is still running the task (doesn't appear to know it's been closed), while the task appears successful from the Prefect Cloud UI.
I think we will use the polling strategy. Thanks for all your help and good call!