Joe D
06/10/2024, 4:50 PMMarvin
06/10/2024, 4:51 PM@timeout
decorator from `prefect`:
python
from prefect import flow, task
from prefect.utilities.tasks import timeout
@task
def long_running_task():
import time
time.sleep(90000) # simulate a long-running task, in seconds
@flow
def my_flow():
with timeout(seconds=86400): # 86400 seconds = 24 hours
long_running_task()
#### 2. Deploy Your Flow to the Fargate WorkPool
Deploy your flow to your ECS WorkPool using the CLI or the Prefect UI:
bash
prefect deployment create -n my-flow-deployment -q my-ecs-work-pool -p my-flow . -ib ecs
#### 3. Configure ECS Task Definition (Optional)
You might want to configure your ECS Task Definition to have a suitable timeout for the task itself, ensuring AWS ECS respects this timeout.
#### Useful Links
- Configuring Work Pools in Prefect Documentation
- Prefect AWS ECS Guide
This approach ensures that any task that exceeds the specified timeout will be terminated appropriately.
### Summary
1. Set a timeout for your flow using @timeout
.
2. Deploy the flow to your ECS WorkPool.
3. Optionally, adjust ECS Task Definition settings to enforce timeouts.
Happy automating, I suppose. If you have additional questions, the Prefect documentation should provide all the answers you ever wanted (or didn't want).Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by