Tom Matthews
06/21/2024, 12:41 PMMarvin
06/21/2024, 12:41 PMrun_deployment
fires up its own infrastructure on Kubernetes rather than running all flows on the same pod as subflows, you need to configure the deployment to use a specific infrastructure block, such as a KubernetesJob
.
### Steps to Configure run_deployment
with Separate Infrastructure:
1. Define the Infrastructure Block:
Ensure that an infrastructure block for KubernetesJob
exists. You can create or configure this block via the Prefect UI or CLI.
2. Specify the Infrastructure Block in Your Deployment YAML:
In your deployment's YAML file, specify the infrastructure type to use.
yaml
version: 2
name: your-flow-deployment
flow_name: your-flow
infra_overrides: []
infrastructure:
type: kubernetes-job
metadata:
# Add your Kubernetes Job configuration here
3. Run the Deployment with Overrides (If necessary):
You might need to provide overrides if deploying via the Prefect CLI.
shell
prefect deployment build path/to/your/flow.py:your_flow_function \
-n your_deployment_name \
--infra-block kubernetes-job/your_kubernetes_block \
-o path/to/your_deployment.yaml
4. Deploy the Flow:
Apply the deployment configuration.
shell
prefect deployment apply path/to/your_deployment.yaml
5. Trigger the Deployment:
When you run the deployment, it will use the specified infrastructure block to fire up its own Kubernetes jobs.
shell
prefect deployment run your-flow/your_deployment_name
For more details, you can refer to the Prefect documentation on deployment configuration and the Prefect GCP worker guide.
Feel free to reach out if you need further assistance.