Let me provide a clear explanation of how process-type workers function, particularly in your self-hosted environment.
Process-type workers operate in a straightforward manner:
1.
Code Source Requirements:
- The flow code must be accessible to the worker
- This can be through various sources:
- Git repositories
- S3 buckets
- Azure Blob Storage
- Google Cloud Storage
- Local filesystem (though this is less common in production)
2.
Dependencies Management:
- Yes, all dependencies required by your flows MUST be installed in the environment where the worker is running
- This means your worker pods (since you're using Helm) need to have:
- All Python packages required by your flows
- Any system dependencies your flows might need
- You can handle this through:
- Custom Docker images with pre-installed dependencies
- Package installation as part of your Helm deployment
- Requirements files specified in your deployment configuration
3.
Worker Operation:
- The worker polls the work pool for new flow runs
- When it finds a run, it:
1. Downloads the flow code from your specified source
2. Creates a new subprocess
3. Executes the flow in that subprocess
4. Monitors the execution and reports status back to your self-hosted Prefect server
4.
Best Practices for Your Setup:
- Since you're using a self-hosted environment with Helm:
- Consider using a base Docker image with common dependencies
- Use consistent Python versions across your worker pods
- Ensure network access to both your PostgreSQL database and code sources
- Configure appropriate resource limits in your Helm values
The process-type worker is one of the simpler worker types, but it's important to ensure your environment is properly configured with all necessary dependencies. Unlike some other worker types (like Docker), it doesn't handle dependency isolation automatically - everything needs to be available in the worker's environment.
You can find more detailed information in the
workers documentation.
Let me store these details about your environment for future reference.