I'm new to Prefect and data engineering. I have a question:
I've created a Python script named "Prefecttest.py" and it's currently stored on my local drive. I've also stored it on Azure Blob Storage cloud. I can successfully run this flow and deploy it. However, I'm facing an issue: when my laptop is turned off, it interrupts the process and the workflow becomes unhealthy. I understand that I might be making a basic mistake, and I'm seeking guidance in the right direction.
My goal is to run this script either from the local drive or from Azure Blob Storage, and I would like it to be scheduled daily via Prefect.
Thank you for your assistance.
08/30/2023, 8:11 AM
you need infrastructure to run your flow - therefore a server on the cloud for example that will be on 24/7 rather than your laptop
prefect does not give the infrastructure you have to run it on your own tin
08/30/2023, 4:48 PM
plus one to redsquare's answer
a common way to get started is to use a major cloud provider like AWS or GCP to get yourself access to a VM where you can run a worker.
let us know if you have other questions!