Tom Matthews
06/21/2024, 12:41 PMMarvin
06/21/2024, 12:41 PMrun_deployment
fires up its own infrastructure on Kubernetes rather than running all flows on the same pod as subflows, you need to configure the deployment to use a specific infrastructure block, such as a KubernetesJob
.
### Steps to Configure run_deployment
with Separate Infrastructure:
1. Define the Infrastructure Block:
Ensure that an infrastructure block for KubernetesJob
exists. You can create or configure this block via the Prefect UI or CLI.
2. Specify the Infrastructure Block in Your Deployment YAML:
In your deployment's YAML file, specify the infrastructure type to use.
yaml
version: 2
name: your-flow-deployment
flow_name: your-flow
infra_overrides: []
infrastructure:
type: kubernetes-job
metadata:
# Add your Kubernetes Job configuration here
3. Run the Deployment with Overrides (If necessary):
You might need to provide overrides if deploying via the Prefect CLI.
shell
prefect deployment build path/to/your/flow.py:your_flow_function \
-n your_deployment_name \
--infra-block kubernetes-job/your_kubernetes_block \
-o path/to/your_deployment.yaml
4. Deploy the Flow:
Apply the deployment configuration.
shell
prefect deployment apply path/to/your_deployment.yaml
5. Trigger the Deployment:
When you run the deployment, it will use the specified infrastructure block to fire up its own Kubernetes jobs.
shell
prefect deployment run your-flow/your_deployment_name
For more details, you can refer to the Prefect documentation on deployment configuration and the Prefect GCP worker guide.
Feel free to reach out if you need further assistance.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by