https://prefect.io logo
Join Slack
Channels
Powered by
Hi Everyone. I am trying to migrate from agents t...
# ask-community
s
Hi Everyone. I am trying to migrate from agents to workers https://docs.prefect.io/latest/guides/upgrade-guide-agents-to-workers/ and I am getting this error in our new worker 
TypeError: object NoneType can't be used in 'await' expression
when I try to submit a job. We are using a KubernetesJob block (and have installed prefect-kubernetes in the prefect-worker), and then created a corresponding work pool like 
k8s_block = KubernetesJob.load(k8s_block_name).publish_as_work_pool()
and the new worker is listening to this pool. When I start a job, it crashes, with the error above. Has anyone had a similar issue when migrating from agents to workers? Any suggestions? Thanks!
As I read more about the difference between workers and agents, and the way that work pools defaults can (and maybe cannot) be overridden by each deployment, I am wondering if our current usage of agents and infra block will migrate to workers at all. For each deployment, we have a kubernetes job yaml file that looks like this:
Copy code
# Base manifest for a prefect job.
apiVersion: batch/v1
kind: Job
metadata:
  labels: {}
spec:
  template:
    spec:
      completions: 1
      containers:
      - env: []
        name: prefect-job
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 1000m
            memory: 1Gi
      nodeSelector:
        type: cpu
      tolerations:
      - key: app
        operator: Equal
        value: job
        effect: NoSchedule
      - key: type
        operator: Equal
        value: shared-worker
        effect: NoSchedule
      parallelism: 1
      restartPolicy: Never
We use this yaml file, along with some other configs to create a k8s_block that essentially represents a Kubernetes Job. In some cases, we may run a sidecar within a job, so those deployments have more complicated yaml files ^. Can I override the entire yaml file that defines the Kubernetes Job so that each of my deployments can be customized in this way? It looks like I may have to parameterize the base job manifest at the work pool level, and then can only override these variables, but not the entire manifest. Is that correct?
@Marvin For a kubernetes work pool, can I override the entire base job manifest for each new deployment?
m
Yes, you can override the entire base job manifest for each new deployment in a Kubernetes work pool. Here's what I found: 1. Prefect Documentation: - You can override job variables on a per-deployment basis either by specifying overrides in the 
deployments.work_pool.job_variables
section of a 
prefect.yaml
file or in the 
job_variables
argument of a Python Flow's 
.deploy()
method. - For advanced customization, you can create work pools with fully customizable job templates, useful for scenarios requiring high customization of the underlying infrastructure. Documentation Source 2. GitHub Issue: - Prefect has updated its worker base job template logic for nested placeholders. This update allows placeholders nested within dictionaries and lists in a JSON structure to be correctly replaced, supporting complex templates for the Kubernetes worker. GitHub Source For more detailed configurations and examples on how to create such advanced customizations, refer to the Kubernetes Worker documentation.
s
@Marvin Can I create job_variables that are themselves dictionaries?
m
Yes, you can create 
job_variables
that are themselves dictionaries in Prefect. Here's what I found: 1. Prefect Documentation: - The 
job_variables
parameter allows passing dictionaries to fine-tune the infrastructure settings for a deployment. You can override environment variables and other settings using dictionaries. - An example from the documentation demonstrates overriding environment variables with a dictionary:
Copy code
python
     get_repo_info.deploy(
         name="my-deployment-never-pull",
         work_pool_name="my-docker-pool",
         job_variables={"env": {"EXTRA_PIP_PACKAGES": "boto3"}},
         image="my-image:my-tag",
         push=False
     )
Documentation Source 2. GitHub Issue: - There was an update allowing for worker 
job_variables
to reference blocks, enabling management and reuse of environment variables across different deployments by referencing predefined blocks stored on the Prefect server. - Example YAML configuration for using `envFrom`:
Copy code
yaml
     deployments:
       - name: "example-deployment"
         work_pool:
           name: "default"
           job_variables:
             image: "my-docker-image:latest"
             envFrom:
               - "{{ prefect.blocks.arbitraryblock.arbitrary-block }}"
GitHub Source These examples show that not only are dictionaries supported as job variables, but they can also be used to reference more complex configurations, including those stored as blocks.
s
@Marvin what is the command to make a work pool from the command line?
m
To create a work pool via the command line using the Prefect CLI, you can use the following command:
Copy code
prefect work-pool create [OPTIONS] NAME
- NAME: This is a required, unique name for the work pool. - OPTIONS: - `--paused`: Creates the work pool in a paused state if provided. - `--type`: Specifies the type of infrastructure that can execute runs from this work pool. - `--set-as-default`: Option to set this work pool as the default. For more details, refer to the Prefect Work Pools Documentation.
s
Looks like this bug fix should resolve my issue: https://github.com/PrefectHQ/prefect/pull/14797. That said, I have installed 2.20.0 and this fix does not seem to show up in the code yet.
18 Views
Open in Slack
PreviousNext
Powered by