@Anna Geller I very much like your article on handling streaming data with prefect. However should I wish to implement same for exchange data that doesn’t run 24/7 but within specific times on week days. Can I use the UI to schedule it in such a manner that the streaming starts and stops within the opening and closing time of the exchange?
02/07/2023, 2:48 PM
Hi Hedgar! you would need to stop the service when you no longer want it to be running and start it again when you do. The easiest might be to do that from the ECS Fargate service console
using this deployment pattern, Prefect only observes your runs, but it doesn't control its remote execution, so scheduling from Prefect UI wouldn't work, but doing the same from AWS console would
02/07/2023, 3:15 PM
Wow, need to study ECS Fargate service closely. Thanks 🙏
@Anna Geller just thinking aloud, if I were to write the data to a parquet file every seconds won’t this operation be expensive?
02/08/2023, 2:42 PM
with Fargate you pay only for a single container that you use which is great because you don't need to have an entire VM or cluster
02/11/2023, 2:32 PM
Cool. From my research, to achieve the above objective that is run during office work hour and stop outside office hours; I may have to schedule task definition. I have checked through the task definition and I can’t find where it can be stopped or start? Any thoughts? @Anna Geller
02/11/2023, 3:07 PM
that's why you need a service - ECS service allows you to run a certain ECS task reliably
you could also just run a container from task definition but this is more for ad-hoc things, not for a service that must run reliably 24/7
02/15/2023, 10:19 PM
@Anna Geller I have been able to get around stoping and starting the stream through manipulating tz and time, I however observed that under infrastructure folder in the ecs_service.yml file, am I suppose to indicate my AWS_ACCOUNT_ID? I already indicated it under actions secrets