I very much like your article on handling str

Question

< Anna Geller> I very much like your article on handling streaming data with prefect However should I wish to implement same for exchange data that doesn t run 24 7 but within specific times on week days Can I use the UI to schedule it in such a manner that the streaming starts and stops within the opening and closing time of the exchange

Anna Geller · Answer

Hi Hedgar you would need to stop the service when you no longer want it to be running and start it again when you do The easiest might be to do that from the ECS Fargate service console

Anna Geller · Answer

using this deployment pattern Prefect only observes your runs but it doesn t control its remote execution so scheduling from Prefect UI wouldn t work but doing the same from AWS console would

Hedgar · Answer

Wow need to study ECS Fargate service closely Thanks pray

Hedgar · Answer

< Anna Geller> just thinking aloud if I were to write the data to a parquet file every seconds won t this operation be expensive

Anna Geller · Answer

with Fargate you pay only for a single container that you use which is great because you don t need to have an entire VM or cluster

Hedgar · Answer

Cool From my research to achieve the above objective that is run during office work hour and stop outside office hours I may have to schedule task definition I have checked through the task definition and I can t find where it can be stopped or start Any thoughts < Anna Geller>

Anna Geller · Answer

that s why you need a service ECS service allows you to run a certain ECS task reliably you could also just run a container from task definition but this is more for ad hoc things not for a service that must run reliably 24 7

Hedgar · Answer

< Anna Geller> I have been able to get around stoping and starting the stream through manipulating tz and time I however observed that under infrastructure folder in the ecs service yml file am I suppose to indicate my AWS ACCOUNT ID I already indicated it under actions secrets