Pooyan
11/27/2023, 7:42 PMprefect_kubernetes
library?Marvin
11/27/2023, 7:43 PMprefect_kubernetes
library, you would typically follow these steps:
1. KubernetesJob infrastructure: This executes flow runs in a Kubernetes Job. It requires kubectl
to be available and you must configure remote Storage. Local storage is not supported for Kubernetes. The ephemeral Prefect API won't work with Docker and Kubernetes. You must have a Prefect server or Prefect Cloud API endpoint set in your agent's configuration.
2. Generate Kubernetes manifest: The Prefect CLI command prefect kubernetes manifest server
automatically generates a Kubernetes manifest with default settings for Prefect deployments. By default, it simply prints out the YAML configuration for a manifest. You can pipe this output to a file of your choice and edit as necessary.
3. Deploy your flows: Ensure your Docker daemon is running first. Deploy all the flows with prefect deploy --all
or deploy them individually by name: prefect deploy -n hello/default
or prefect deploy -n hello/arthur
.
4. Run the flows: Once the deployments are successfully created, you can run them from the UI or the CLI: prefect deployment run hello/default
prefect deployment run hello/arthur
For more detailed information, you can refer to the Prefect documentation here and here.
However, it's important to note that these documents do not specifically mention Jinja templating. If you want to use Jinja templating in your Kubernetes job YAML files, you would typically use it to dynamically generate your YAML files based on certain variables or conditions. You would then use the prefect_kubernetes
library to interact with these dynamically generated YAML files.