Hi guys I have been struggling for the past few months to ge Prefect Community #ask-community

Hi guys! I have been struggling for the past few m...

Jacob Crabtree

07/17/2024, 1:30 PM

Hi guys! I have been struggling for the past few months to get our system upgraded to a newer prefect version. We are still using the old infrastructure block/agent based method and we would like to transition to the new worker based deployments. I keep running into errors like this when I try adding my old infrastructure block settings as job variables:

Copy code

prefect.deployments.runner.DeploymentApplyError: Error creating deployment: <ValidationError: "False is not of type 'string'">

Does anyone have a simple reference for how to use a job variables section/base job template to deploy a flow using the python SDK?

Jacob Crabtree

07/17/2024, 1:32 PM

For reference, I have tried adding the settings to the work pool like this:

Copy code

base_job_template = {
  "variables": {},
  "job_configuration": {
    "image_pull_policy": "IfNotPresent",
    "volumes": [
      "/home/pipeline:/home/pipeline",
      "/sizzle_storage:/sizzle_storage"
    ],
    "labels": {
      "app": "prefect-flow-runs",
      "environment": "review",
      "commit": "56e7faa6",
      "logging": "promtail",
      "logging_jobname": "containerlogs",
      "flow": "add-discord-roles-to-user",
      "deployment": "new_streamer-1",
      "work_pool": "new-streamer"
    },
    "env": {
      "SIZZLE_WORKFLOWS_DATA_DIR": "/sizzle_storage/AIME_1/prefect_review",
      "PREFECT_TASKS_REFRESH_CACHE": true,
      "VAULT_APPROLE_SECRET_ID": "[MASKED]",
      "SIZZLE_SLACK_NOTIFY": false,
      "MONGO_DATABASE_NAME": "sizzleProduction_V1",
      "WORKFLOW_ENV_FILE": ".env.review",
      "SIZZLE_ENVIRONMENT": "review",
      "CI_COMMIT_REF_SLUG": "build-prefect-upgrade"
    },
    "auto_remove": true,
    "stream_output": true,
    "network_mode": "host"
  }
}

await client.create_work_pool(
	WorkPoolCreate(
	    name=docker_deployment.work_pool_name,
	    type="docker",
	    base_job_template=base_job_template,
	)
)

And I have tried adding them to the

flow.deploy

method like this:

Copy code

job_variables = {
  "volumes": [
    "/home/pipeline:/home/pipeline",
    "/sizzle_storage:/sizzle_storage"
  ],
  "labels": {
    "app": "prefect-flow-runs",
    "environment": "review",
    "commit": "56e7faa6",
    "logging": "promtail",
    "logging_jobname": "containerlogs",
    "flow": "add-discord-roles-to-user",
    "deployment": "new_streamer-1",
    "work_pool": "new-streamer"
  },
  "env": {
    "SIZZLE_WORKFLOWS_DATA_DIR": "/sizzle_storage/AIME_1/prefect_review",
    "PREFECT_TASKS_REFRESH_CACHE": true,
    "VAULT_APPROLE_SECRET_ID": "[MASKED]",
    "SIZZLE_SLACK_NOTIFY": false,
    "MONGO_DATABASE_NAME": "sizzleProduction_V1",
    "WORKFLOW_ENV_FILE": ".env.review",
    "SIZZLE_ENVIRONMENT": "review",
    "CI_COMMIT_REF_SLUG": "build-prefect-upgrade"
  },
  "auto_remove": true,
  "stream_output": true,
  "network_mode": "host"
}

await docker_deployment.flow.deploy(
    name=deployment_name,
    work_pool_name=docker_deployment.work_pool_name,
    tags=docker_deployment.tags + default_deployment_tags,
    image=docker_image_url,
    job_variables=job_variables,
    build=False,
    push=False,
)

2 Views

Open in Slack

Previous Next