I'm glad I didn't give up on myself, I think I have started wrapping my head around this prefect thing anyway thanks to https://medium.com/the-prefect-blog/getting-started-with-prefect-flows-98e8881ffaf9 • I have signed up to prefect cloud • Changed backend to cloud • Registered my project and flow • Done

prefect agent local start

• Was also able to click the

quick run

and I got a beautiful green bar of success! However I have a challenge: My AWS ec2 with the help of lambda function get started at certain time of the day and shuts down at certain time of the day. How do I ensure that my flow runs when AWS ec2 starts for the day? Do I need to create a bash script that start the local agent via a crontab on the remote instance?

If your EC2 instance will start 5 minutes before the flow starts, then you just need to have the prefect agent start on the EC2 instance startup. That way, the agent will be listening for prefect cloud to tell it when to run the flow, and hence it will run the flow as scheduled. That is, the scheduling is handled by prefect cloud, as per the schedule provided to prefect cloud when you register the flow.

Upon starting the instance • pipenv shell #activate virtual env •

Perfect register --project my project -p myflow.py -n ‘firstflow’

•

prefect agent local start

How do I know,though schedule time is not now,that this took effect using my cloud dashboard especially as I still see “last agent queried 2 hours ago”under local agents.

To ensure that your agent restarts during unexpected infrastructure issues, you can use Supervisor - this docs page explains that. In general, Prefect agent is a lightweight process that is supposed to be always-on and running 24/7 - if having a separate instance for that seems to expensive in your use case, perhaps you can check that guide to run a single container service for Prefect agent on ECS Fargate? Having said that, there are some users who did the same setup as you suggested having a Lambda function scheduling instance startup and shutdown: https://discourse.prefect.io/t/building-a-data-lake-with-prefect-cloud-s3-glue-and-athena-by-kat-hempstalk/382