Hi everyone, I'm using Prefect 3. Currently, here's my set-up: • Prefect Cloud Server • Self-hosting worker on EC2, where I create a work pool and start worker • Flow code in Git ◦ one CICD job to build and push docker image to ECR, as well as create deployment in Prefect Server ◦ another CICD job to trigger deployment run Currently, the above set-up works. However, after finishing a run, the worker in the work pool becomes inactive. So I have to go into EC2 and start the worker again for subsequent runs to work. Does anyone know why the worker will become inactive? I was under the impression it will stay active and continuously poll the work pool for new jobs (especially with a constantly running ec2)

Hi QTai, my guess is that your worker process is terminating due to resource contention or some external service. You'll need to daemonize the worker to make sure that it stays up and running.

@Bianca Hoch Got it Bianca, thanks a lot! For the above set-up, I'm using a simple test script. However, my intended use case is for training of various Machine Learning models, where some may require more resources like GPU. I think it will be very costly to keep an ec2 GPU instance running and polling the Prefect Server. Ideally, when the request comes in, depending on the job, the right amount of resource and the worker will be spinned up. The worker executes the run, and then gets terminated after to save cost. It seems that using a push work pool and ECS with different task definitions might work. However, if i'm not wrong, push work pool only works with Prefect Cloud. We may need to set this up on-prem, so we'll need to self-host the Prefect Server and need to use an ECS alternative like Kubernetes. Do you have any recommendations for this?

Hmmm..if you're self-hosting, I think standing up an ECS worker as an ECS service, or spinning up your own EKS cluster with a Kubernetes worker would be your best bet to benefit from dynamic infrastructure.