Asif Imran
02/23/2021, 5:19 PMSensors
from airflow. I see this helpful link[1] from Jeremiah -- essentially do the poke-sleep-poke yourself iiuc. Any changes since his post? Recently AF introduced the notion of SmartSensors[2] which removes a fair bit of work duplication (its fairly typical for me to have several workflows all polling on the same S3 success file). I have similar worries that such polling will hog up resources (e.g going over the resources in my ecs cluster)
[1] https://prefect-community.slack.com/archives/CL09KU1K7/p1602088074097600?thread_ts=1602087747.097500&cid=CL09KU1K7
[2] https://airflow.apache.org/docs/apache-airflow/stable/smart-sensor.htmlJenny
02/23/2021, 9:34 PMAsif Imran
02/23/2021, 9:36 PMAsif Imran
02/24/2021, 5:24 PMPrefect doesn't really have a concept of "worker" in the way that Airflow does - the only cost to waiting / sleeping is a process or job in your environment, and there's no competition from other tasks for this process / job