Receiving a SIGTERM signal in a Prefect workflow can be attributed to several causes:
1.
Cancellation Requests: Manually cancelling a flow run via the UI sends a SIGTERM to the running process to stop it.
- Refer to the GitHub Issue:
Unable to handle 'Cancelling' events.
2.
Resource Management: If the workflow exceeds its allocated resources (e.g., CPU/memory limits), the orchestrator (like Kubernetes) might terminate it.
3.
System Reboots: Workflows may receive a SIGTERM if the underlying system is set to reboot, often observed in maintenance windows.
4.
Shut Down Commands: Dask or other computing clusters might not properly communicate shutdowns, leading to orphaned pods.
- More details can be found in this GitHub Issue:
Flow cancellation does not delete Dask Scheduler & Worker pods.
5.
Improper File Handling: Not properly closing resources like file handles before abrupt terminations can lead to data corruption and unexpected behavior during terminations.
- Details in the GitHub Issue:
Unable to handle 'Cancelling' events.
These are some common causes, but specific scenarios can vary based on workflow configuration and execution environment.
For more detailed examples and contexts, you can browse the linked GitHub issues.