@Marvin, show me please example of base job template for Docker work pool type, where custom envvars, image and volumes are customized

Prefect's documentation outlines how to customize a Docker work pool base job template, including the customization of environment variables, Docker image, and volumes. Here is an example of how to define a base job template for Docker work pool with those customizations: ### Base Job Template

json
{
  "job_configuration": {
    "image": "{{ image }}",
    "env": [
      {
        "name": "MY_CUSTOM_ENV",
        "value": "{{ env.MY_CUSTOM_ENV }}"
      }
    ],
    "volumes": [
      {
        "host_path": "/host/path",
        "container_path": "/container/path"
      }
    ]
  },
  "variables": {
    "type": "object",
    "properties": {
      "image": {
        "type": "string",
        "default": "my-custom-image:latest"
      },
      "env": {
        "type": "object",
        "properties": {
          "MY_CUSTOM_ENV": {
            "type": "string",
            "default": "default_value"
          }
        }
      },
      "volumes": {
        "type": "array",
        "default": [{
          "host_path": "/default/host/path",
          "container_path": "/default/container/path"
        }]
      }
    }
  }
}

### Update Command To update the work pool with this base job template:

bash
prefect work-pool update --base-job-template base-job-template.json my-work-pool

### Explanation 1. job_configuration: - image: The Docker image to use. - env: Custom environment variables. - volumes: Host to container volume mappings. 2. variables: - Defines default values and types for

image

,

env

, and

volumes

. - Uses OpenAPI specification to declare variable properties. For in-depth details and more examples, you can refer to Prefect's Work Pools documentation.

@Marvin, it doesn’t work. i saved example you provide me to job-template.json file and run following command: prefect work-pool create test2 --type docker --base-job-template ./job-template.json i get following error: prefect.exceptions.PrefectHTTPStatusError: Client error ‘422 Unprocessable Entity’ for url ‘http://host.docker.internal:4200/api/work_pools/’ Response: {‘exception_message’: ‘Invalid request received.’, ‘exception_detail’: [{‘loc’: [‘body’, ‘base_job_template’], ‘msg’: ‘The variables specified in the job configuration template must be present as properties in the variables schema. Your job configuration uses the following undeclared variable(s): env.MY_CUSTOM_ENV.’, ‘type’: ‘value_error’}], ‘request_body’: {‘name’: ‘test2’, ‘type’: ‘docker’, ‘base_job_template’: {‘job_configuration’: {‘image’: ‘{{ image }}’, ‘env’: [{‘name’: ‘MY_CUSTOM_ENV’, ‘value’: ‘{{ env.MY_CUSTOM_ENV }}’}], ‘volumes’: [{‘host_path’: ‘/host/path’, ‘container_path’: ‘/container/path’}]}, ‘variables’: {‘type’: ‘object’, ‘properties’: {‘image’: {‘type’: ‘string’, ‘default’: ‘my-custom-image:latest’}, ‘env’: {‘type’: ‘object’, ‘properties’: {‘MY_CUSTOM_ENV’: {‘type’: ‘string’, ‘default’: ‘default_value’}}}, ‘volumes’: {‘type’: ‘array’, ‘default’: [{‘host_path’: ‘/default/host/path’, ‘container_path’: ‘/default/container/path’}]}}}}, ‘is_paused’: False}}

Your message was too long by 84 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

, it doesn’t work. i saved example you provide me to job-template.json file and run following command:

prefect work-pool create test2 --type docker --base-job-template ./job-template.json i get following error: prefect.exceptions.PrefectHTTPStatusError: Client error ‘422 Unprocessable Entity’ for url ‘http://host.docker.internal:4200/api/work_pools/’ Response: {‘exception_message’: ‘Invalid request received.’, ‘exception_detail’: [{‘loc’: [‘body’, ‘base_job_template’], ‘msg’: ‘The variables specified in the job configuration template must be present as properties in the variables schema. Your job configuration uses the following undeclared variable(s): env.MY_CUSTOM_ENV.’, ‘type’: ‘value_error’}], ‘request_body’: {‘name’: ‘test2’, ‘type’: ‘docker’, ‘base_job_template’: {‘job_configuration’: {‘image’: ‘{{ image }}’, ‘env’: [{‘name’: ‘MY_CUSTOM_ENV’, ‘value’: ‘{{ env.MY_CUSTOM_ENV }}’}], ‘volumes’: [{‘host_path’: ‘/host/path’, ‘container_path’: ‘/container/path’}]}, ‘variables’: {‘type’: ‘object’, ‘properties’: {‘image’: {‘type’: ‘string’, ‘default’: ‘my-custom-image:

@Marvin i saved example to job-template.json file and run: prefect work-pool create test2 --type docker --base-job-template ./job-template.json i get following error: prefect.exceptions.PrefectHTTPStatusError: Client error ‘422 Unprocessable Entity’ for url ‘http://host.docker.internal:4200/api/work_pools/’ Response: {‘exception_message’: ‘Invalid request received.’, ‘exception_detail’: [{‘loc’: [‘body’, ‘base_job_template’], ‘msg’: ‘The variables specified in the job configuration template must be present as properties in the variables schema. Your job configuration uses the following undeclared variable(s): env.MY_CUSTOM_ENV.’, ‘type’: ‘value_error’}], ‘request_body’: {‘name’: ‘test2’, ‘type’: ‘docker’, ‘base_job_template’: {‘job_configuration’: {‘image’: ‘{{ image }}’, ‘env’: [{‘name’: ‘MY_CUSTOM_ENV’, ‘value’: ‘{{ env.MY_CUSTOM_ENV }}’}], ‘volumes’: [{‘host_path’: ‘/host/path’, ‘container_path’: ‘/container/path’}]}, ‘variables’: {‘type’: ‘object’, ‘properties’: {‘image’: {‘type’: ‘string’, ‘default’: ‘my-custom-image:latest’}, ‘env’: {‘type’: ‘object’, ‘properties’: {‘MY_CUSTOM_ENV’: {‘type’: ‘string’, ‘default’: ‘default_value’}}}, ‘volumes’: {‘type’: ‘array’, ‘default’: [{‘host_path’: ‘/default/host/path’, ‘container_path’: ‘/default/container/path’}]}}}}, ‘is_paused’: False}}

Your message was too long by 73 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

i saved example to job-template.json file and run:

prefect work-pool create test2 --type docker --base-job-template ./job-template.json i get following error: prefect.exceptions.PrefectHTTPStatusError: Client error ‘422 Unprocessable Entity’ for url ‘http://host.docker.internal:4200/api/work_pools/’ Response: {‘exception_message’: ‘Invalid request received.’, ‘exception_detail’: [{‘loc’: [‘body’, ‘base_job_template’], ‘msg’: ‘The variables specified in the job configuration template must be present as properties in the variables schema. Your job configuration uses the following undeclared variable(s): env.MY_CUSTOM_ENV.’, ‘type’: ‘value_error’}], ‘request_body’: {‘name’: ‘test2’, ‘type’: ‘docker’, ‘base_job_template’: {‘job_configuration’: {‘image’: ‘{{ image }}’, ‘env’: [{‘name’: ‘MY_CUSTOM_ENV’, ‘value’: ‘{{ env.MY_CUSTOM_ENV }}’}], ‘volumes’: [{‘host_path’: ‘/host/path’, ‘container_path’: ‘/container/path’}]}, ‘variables’: {‘type’: ‘object’, ‘properties’: {‘image’: {‘type’: ‘string’, ‘default’: ‘my-custom-image:latest’}, ‘env’: {‘type’: ‘

@Marvin i saved example to job-template.json file and run: prefect work-pool create test2 --type docker --base-job-template ./job-template.json i get following error: prefect.exceptions.PrefectHTTPStatusError: Client error ‘422 Unprocessable Entity’ for url ‘http://host.docker.internal:4200/api/work_pools/’ Response: {‘exception_message’: ‘Invalid request received.’, ‘exception_detail’: [{‘loc’: [‘body’, ‘base_job_template’], ‘msg’: ‘The variables specified in the job configuration template must be present as properties in the variables schema. Your job configuration uses the following undeclared variable(s): env.MY_CUSTOM_ENV.’, ‘type’: ‘value_error’}],….

The error you're seeing is due to the use of variables in the

job_configuration

section that haven't been properly declared in the

variables

section. Based on the issue #12201, it appears that variables should be prefixed with

variables.

. ### Corrected Base Job Template Here's the corrected version of your `job-template.json`:

json
{
  "job_configuration": {
    "image": "{{ variables.image }}",
    "env": [
      {
        "name": "MY_CUSTOM_ENV",
        "value": "{{ variables.env.MY_CUSTOM_ENV }}"
      }
    ],
    "volumes": [
      {
        "host_path": "{{ variables.volumes[0].host_path }}",
        "container_path": "{{ variables.volumes[0].container_path }}"
      }
    ]
  },
  "variables": {
    "type": "object",
    "properties": {
      "image": {
        "type": "string",
        "default": "my-custom-image:latest"
      },
      "env": {
        "type": "object",
        "properties": {
          "MY_CUSTOM_ENV": {
            "type": "string",
            "default": "default_value"
          }
        }
      },
      "volumes": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "host_path": {
              "type": "string",
              "default": "/default/host/path"
            },
            "container_path": {
              "type": "string",
              "default": "/default/container/path"
            }
          }
        },
        "default": [{
          "host_path": "/default/host/path",
          "container_path": "/default/container/path"
        }]
      }
    }
  }
}

### Run the Update Command Again Retry updating your work pool using this corrected template:

bash
prefect work-pool create test2 --type docker --base-job-template ./job-template.json

### Documentation and Resources For further details, you can explore: - Prefect Work Pools Documentation - The related GitHub issue #12201 This should resolve the variable declaration problem you're encountering. Let me know if you need any further assistance.

it reports still same issue: The variables specified in the job configuration template must be present as properties in the variables schema. Your job configuration uses the following undeclared variable(s): variables.env.MY_CUSTOM_ENV ,variables.volumes[0].host_path ,variables.volumes[0].container_path ,variables.image.

It appears I made a mistake in the handling of nested properties. Based on the error message, it seems that the variables must be declared correctly and referenced directly. I'll correct the structure to avoid using nested references. ### Corrected Base Job Template

json
{
  "job_configuration": {
    "image": "{{ variables.image }}",
    "env": [
      {
        "name": "MY_CUSTOM_ENV",
        "value": "{{ variables.MY_CUSTOM_ENV }}"
      }
    ],
    "volumes": [
      {
        "host_path": "{{ variables.host_path }}",
        "container_path": "{{ variables.container_path }}"
      }
    ]
  },
  "variables": {
    "type": "object",
    "properties": {
      "image": {
        "type": "string",
        "default": "my-custom-image:latest"
      },
      "MY_CUSTOM_ENV": {
        "type": "string",
        "default": "default_value"
      },
      "host_path": {
        "type": "string",
        "default": "/default/host/path"
      },
      "container_path": {
        "type": "string",
        "default": "/default/container/path"
      }
    }
  }
}

### Run the Update Command Again Retry updating your work pool using this corrected template:

bash
prefect work-pool create test2 --type docker --base-job-template ./job-template.json

This approach avoids nested properties and directly declares the variables in the

variables

section, making them easier to reference in

job_configuration

. Let me know if this resolves the issue.

ok, i can confirm that only way how to do it is copy JSON structure from webadmin UI (on URL

<http://localhost:4200/work-pools/work-pool/><pool-name>/edit

, scroll down to “Base Job Template” and click on “Advanced”) and change “default” values in individual sections inside “variables/properties” items