Yes, it's absolutely possible to set CPU and memory resources at the deployment level rather than the work pool level when using Kubernetes workers. There are a couple of ways to achieve this: 1. Using the UI: - Navigate to the deployment's configuration - Go to the 'Advanced' tab - Edit the job template to specify custom resource requirements 2. Using Python:

python
flow.deploy(
    name="my-deployment",
    work_pool_name="my-k8s-pool",
    job_variables={
        "job_configuration": {
            "spec": {
                "template": {
                    "spec": {
                        "containers": [{
                            "resources": {
                                "requests": {
                                    "cpu": "500m",
                                    "memory": "512Mi"
                                },
                                "limits": {
                                    "cpu": "1000m",
                                    "memory": "1Gi"
                                }
                            }
                        }]
                    }
                }
            }
        }
    }
)

3. Using prefect.yaml:

yaml
deployments:
  - name: my-deployment
    work_pool:
      name: my-k8s-pool
    job_variables:
      job_configuration:
        spec:
          template:
            spec:
              containers:
                - resources:
                    requests:
                      cpu: "500m"
                      memory: "512Mi"
                    limits:
                      cpu: "1000m"
                      memory: "1Gi"

These deployment-level configurations will override any default resource settings specified at the work pool level. This gives you the flexibility to have different resource requirements for different flows, even when they're running on the same work pool. For more details on customizing Kubernetes deployments and managing work pools, you can refer to the documentation: - Kubernetes Infrastructure Guide - Managing Work Pools - Customizing Job Variables

For Kubernetes-based work pools specifically, there are several ways to check the resources your flows are running with: 1. Using kubectl:

bash
   # List all pods with their resource requests and limits
   kubectl get pods -n your-namespace
   
   # Get detailed information about a specific pod
   kubectl describe pod <pod-name> -n your-namespace

This will show you the actual resource requests and limits configured for the pod running your flow. 2. Using Kubernetes Dashboard: If you have the Kubernetes Dashboard installed, you can: - Navigate to the namespace where your flows are running - View the pods section - Check the resource allocation and usage metrics visually 3. Using the Prefect UI: - Navigate to the flow run - Check the logs section - Look for the infrastructure details, which will show you the worker and infrastructure being used 4. Using Kubernetes Metrics:

bash
   # If metrics-server is installed
   kubectl top pod <pod-name> -n your-namespace

This shows actual resource consumption of your running pods. 5. Programmatically during flow execution: You can add logging within your flow to print the available resources:

python
   from prefect import flow
   import os

   @flow
   def my_flow():
       # Print CPU limit from cgroups (if available)
       try:
           with open('/sys/fs/cgroup/cpu/cpu.cfs_quota_us', 'r') as f:
               cpu_quota = int(f.read())
           with open('/sys/fs/cgroup/cpu/cpu.cfs_period_us', 'r') as f:
               cpu_period = int(f.read())
           cpu_cores = cpu_quota / cpu_period
           print(f"CPU cores allocated: {cpu_cores}")
       except:
           print("Could not read CPU information")

       # Print memory limit
       try:
           with open('/sys/fs/cgroup/memory/memory.limit_in_bytes', 'r') as f:
               memory_bytes = int(f.read())
           print(f"Memory limit: {memory_bytes / (1024*1024*1024):.2f}GB")
       except:
           print("Could not read memory information")

For more comprehensive monitoring, you might want to consider: 1. Prometheus and Grafana: - Set up Prometheus to scrape Kubernetes metrics - Create Grafana dashboards to visualize resource usage over time - Tag your Prefect flow pods appropriately to make them easily queryable 2. Cloud Provider Tools: If you're running on a cloud provider: - AWS CloudWatch (for EKS) - Azure Monitor (for AKS) - Google Cloud Monitoring (for GKE) These tools can provide more detailed metrics and historical data about your flow runs' resource usage. Remember that the actual resource usage might differ from the requested/limited resources, and it's good practice to monitor both to ensure your flows have the resources they need without over-provisioning. For more information about work pools and infrastructure monitoring, you can refer to: - Work Pools Concepts - Work Pools and Workers Guide