Hello Prefect Community 🙂 I am relatively new to Prefect and am seeking advice on how best to structure my process. To provide some context, I aim to run Prefect within a Docker container, paired with a suitable database, all set up on my server. I have my crawlers/flows ready, and have created a Dockerfile as well as a Docker Compose.
My intention is to manage and schedule these containers/flows through Prefect. More specifically, I want Prefect to initiate the respective Docker container at certain times, execute the flow, and then terminate the container. All these 'Flow' containers will be operating on the same server.
My main question is about the deployment process: How should I proceed? I am specifically interested to learn from anyone who has had a similar scenario and has created a setup accordingly.
I am open to feedback and would greatly appreciate your support in this learning process. Feel free to share your tips, experiences, or any resources that could be beneficial for my use-case. Thank you in advance for your time and consideration.