< Marvin> how should I use `job variables` in the `Create Fl Prefect Community #ask-marvin

<@ULVA73B9P>, how should I use `job_variables` in ...

Martijn van Elferen

06/17/2024, 6:15 PM

@Marvin, how should I use

job_variables

in the

Create Flow Run From Deployment

API to override?

Marvin

06/17/2024, 6:15 PM

To override

job_variables

in the

Create Flow Run from Deployment

API, you can pass in a dictionary of variables that you want to override. Here's a brief walkthrough based on the information gathered: ### Documentation Excerpts 1. Overriding Job Variables in Deployments: - You can pass in a dictionary of variables into your flow run infrastructure. - Example syntax for passing job variables via CLI:

Copy code

shell
      prefect deployment run \
      --id "<deployment_id>" \
      --job-variable MY_NEW_ENV_VAR=42 \
      --job-variable HELLO=THERE

- This can also be applied in automation actions with Jinja templates. 2. Setting Up Job Variables in Deployment Script: - In your deployment script, you can define job variables directly in the

deploy()

function. - Example:

Copy code

python
      flow.from_source(
          source="<https://github.com/your-repo.git>",
          entrypoint="path/to/flow.py:flow_function"
      ).deploy(
          name="your-deployment",
          work_pool_name="your-work-pool",
          job_variables={
              "env": {
                  "EXECUTION_ENVIRONMENT": "staging",
                  "MY_NOT_SO_SECRET_CONFIG": "config_value"
              }
          }
      )

Make sure the environment variables are set on the machine running the script. ### GitHub Issues A search through GitHub issues didn't yield specific examples directly relating to using

job_variables

in API calls. The consistent advice is to ensure environment variables are correctly set and parsed in your script or deployment configuration. ### Links to Sources - Overriding Job Variables in Prefect Deployments - Prefect Documentation - Prefect Deployments Guide - Prefect Documentation ### Summary To override job variables using the API, pass a dictionary of the variables you need to override in your deployment script or API call. Ensure all needed environment variables and configurations are correctly set and referenced.

Martijn van Elferen

06/17/2024, 6:16 PM

@Marvin, if a deployment is triggered using the

Create Flow Run From Deployment

API, is it still possible to override the Work Pool's configuration?

Marvin

06/17/2024, 6:16 PM

Yes, it is possible to override a Work Pool's configuration when triggering a deployment using the

Create Flow Run From Deployment

API. Here are the detailed steps on how to accomplish it: ### Steps to Override Work Pool Configuration 1. Using
infra_overrides
: - You can use the

infra_overrides

parameter to specify overrides for the Work Pool's infrastructure. - Example usage:

Copy code

python
      from prefect.deployments import Deployment
      from prefect import flow, task

      @task
      def print_hello(name):
          print(f"Hello {name}!")

      @flow(name="Hello Flow")
      def hello_world(name="world"):
          print_hello(name)

      # Define your overrides
      infra_overrides = {
          "image": "python:3.11.6-slim",
          "env": {
              "EXTRA_PIP_PACKAGES": "s3fs"
          }
      }

      # Build and apply the deployment with infra_overrides
      Deployment.build_from_flow(
          hello_world,
          name="demo-deployment",
          work_pool_name="your-work-pool",
          infra_overrides=infra_overrides,
          apply=True
      )

This script sets and overrides specific infrastructure-related parameters for a deployment. 2. Using Environment Variables and CLI: - In the CLI, you can pass job variables using flags to override work pool configurations. - Command example:

Copy code

sh
      prefect deployment run --id "<deployment_id>" --job-variable MY_NEW_ENV_VAR=42 --job-variable HELLO=THERE

### Documentation and Issues - According to the Prefect documentation, custom runs allow you to pass a dictionary of variables to the flow run infrastructure. This also applies to automations kicked off via actions. - The GitHub issues search revealed some bugs and fixes about respecting certain

infra_overrides

and ensuring valid variables. For example, issue 10931 discusses respecting all infra overrides and appropriate error checks for invalid variables. ### Summary It’s clear that, whether using the Prefect API, CLI, or the deployment script, you have multiple ways to override work pool configurations dynamically. ### Useful Links - Overriding Job Variables in Prefect Deployments - Work Pools Concept in Prefect - Issue 10931 on GitHub Let me know if you need more examples or a specific scenario covered.

Martijn van Elferen

06/17/2024, 6:18 PM

@Marvin,

infra_overrides

is deprecated, how to use it now for this API call: https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{id}/create_flow_run

Marvin

06/17/2024, 6:18 PM

To override Work Pool configurations when triggering a deployment using the

Create Flow Run From Deployment

API, since

infra_overrides

is deprecated, you will need to use the

parameters

field in the API call to provide any necessary configuration overrides. Here’s a step-by-step guide on how you might proceed: ### API Endpoint

POST <https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{id}/create_flow_run>

### Example Usage To override Work Pool configurations when creating a flow run, specify any overrides within the

parameters

field of your API request body. Here’s an example showing how you might structure the request:

Copy code

json
{
  "name": "Your Flow Run Name",
  "parameters": {
    "job_variables": {
      "image": "python:3.8",
      "env": {
        "NEW_ENV_VAR": "value",
        "ANOTHER_ENV_VAR": "value"
      }
    }
  },
  "tags": ["optional-tag"],
  "state_name": "Scheduled"
}

### Making the Request In Python, an example API request using the

requests

library could look like this:

Copy code

python
import requests

account_id = "your_account_id"
workspace_id = "your_workspace_id"
deployment_id = "your_deployment_id"

url = f"<https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{deployment_id}/create_flow_run>"

headers = {
    "Authorization": "Bearer YOUR_PREFECT_API_KEY",
    "Content-Type": "application/json"
}

payload = {
  "name": "Your Flow Run Name",
  "parameters": {
    "job_variables": {
      "image": "python:3.8",
      "env": {
        "NEW_ENV_VAR": "value",
        "ANOTHER_ENV_VAR": "value"
      }
    }
  },
  "tags": ["optional-tag"],
  "state_name": "Scheduled"
}

response = <http://requests.post|requests.post>(url, headers=headers, json=payload)

if response.status_code == 201:
    print("Flow Run created successfully!")
else:
    print(f"Failed to create Flow Run: {response.text}")

### Documentation References For further reference, you might find the following links useful: - Prefect API Reference - Work Pools Concept - Deployments Overview This approach lets you customize the work pool's configuration while creating a flow run, addressing the deprecation of

infra_overrides

Martijn van Elferen

06/19/2024, 8:54 AM

@Marvin, On a deployment, in the configuration i set the job variables to: "

Copy code

{
  "image": "{{ $GCR_IMAGE }}"
}

" Now I am expecting that when I call a flow, and set the Job Variables to: "job_variables": { "GCR_IMAGE": "eu.gcr.io/abc/prefect/prefect-pricing:production" " that the deployment will pull the correct image, but instead it gives the following error. 'Expected an image path like [host/]repo-path[:tag and/or @digest], where host is one of [region.]gcr.io, [region-]docker.pkg.dev or docker.io but obtained {{ $GCR_IMAGE }}'}]}]">

Marvin

06/19/2024, 8:54 AM

The issue arises because the

GCR_IMAGE

variable in the job variables is not being substituted correctly in the deployment configuration. The provided error indicates that the template variable

{{ $GCR_IMAGE }}

is not replaced with the actual image path at runtime. ### Correct Usage of Template Variables Based on the information gathered, here are a few key points to ensure your template variables are correctly substituted: 1. Variable Substitution in Deployment Configuration: Ensure that the template variable

{{ $GCR_IMAGE }}

is correctly defined in your deployment configuration and that it's substitutable. 2. Proper Template Declaration: Ensure that in your job variables, you correctly set

GCR_IMAGE

to the desired value. ### Updated Deployment Configuration and Flow Run Call #### Deployment Configuration Ensure your deployment configuration has the job variable correctly set as a template:

Copy code

yaml
deployments:
  - name: demo-deployment
    entrypoint: demo_project/demo_flow.py:some_work
    work_pool:
      name: local
    job_variables:
      image: "{{ $GCR_IMAGE }}"

#### Creating Flow Run with Job Variables When creating a flow run, ensure you pass the job variable like this:

Copy code

json
{
  "name": "Your Flow Run Name",
  "parameters": {
    "job_variables": {
      "GCR_IMAGE": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
    }
  },
  "tags": ["optional-tag"],
  "state_name": "Scheduled"
}

### Example API Call with Python Here’s how you might make this API call using Python and the

requests

library:

Copy code

python
import requests

account_id = "your_account_id"
workspace_id = "your_workspace_id"
deployment_id = "your_deployment_id"
api_key = "YOUR_PREFECT_API_KEY"

url = f"<https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{deployment_id}/create_flow_run>"

headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

payload = {
  "name": "Your Flow Run Name",
  "parameters": {
    "job_variables": {
      "GCR_IMAGE": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
    }
  },
  "tags": ["optional-tag"],
  "state_name": "Scheduled"
}

response = <http://requests.post|requests.post>(url, headers=headers, json=payload)

if response.status_code == 201:
    print("Flow Run created successfully!")
else:
    print(f"Failed to create Flow Run: {response.text}")

### Additional Resources - Overriding Work Pool Job Variables explains setting job variables in deployments. - Deployments Guide for more details on deployment configurations. This should resolve the substitution issue and ensure your deployment uses the correct image path.

Martijn van Elferen

06/19/2024, 9:06 AM

@Marvin, if my deployment in prefect.yaml looks like:

Copy code

deployments:
- # base metadata
    name: null
    version: "{{ build-image.tag }}"
    tags:
        - "{{ $my_deployment_tag }}"
        - "{{ prefect.variables.some_common_tag }}"
    description: null
    schedule: null

    # flow-specific fields
    entrypoint: null
    parameters: {}

    # infra-specific fields
    work_pool:
        name: "my-k8s-work-pool"
        work_queue_name: null
        job_variables:
            image: "{{ build-image.image }}"
            cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-favorite-config }}"

How do I send the value for

Copy code

image: "{{ build-image.image }}"

in an API body? Like this? payload = { "state": { "type": "SCHEDULED" }, "job_variables": { "build-image": { "image": "eu.gcr.io/abc/prefect/prefect-pricing:production", } } }?

Marvin

06/19/2024, 9:06 AM

Your intention to set the job variables, particularly

image

, through the API payload is correct. However, you need to ensure that the

job_variables

are correctly structured in the payload you send to the API. Here’s how you can structure the API payload to set the

GCR_IMAGE

variable correctly:

Copy code

json
{
  "state": {
    "type": "SCHEDULED"
  },
  "parameters": {
    "job_variables": {
      "build-image.image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
    }
  }
}

In this payload,

job_variables

should be part of the

parameters

field, and the template variable name

build-image.image

needs to correspond exactly to how it’s referenced in your

prefect.yaml

. ### Complete Example API Call Here’s a complete example using Python and the

requests

library:

Copy code

python
import requests

account_id = "your_account_id"
workspace_id = "your_workspace_id"
deployment_id = "your_deployment_id"
api_key = "YOUR_PREFECT_API_KEY"

url = f"<https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{deployment_id}/create_flow_run>"

headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

payload = {
    "state": {
        "type": "SCHEDULED"
    },
    "parameters": {
        "job_variables": {
            "build-image.image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
        }
    }
}

response = <http://requests.post|requests.post>(url, headers=headers, json=payload)

if response.status_code == 201:
    print("Flow Run created successfully!")
else:
    print(f"Failed to create Flow Run: {response.text}")

### Notes 1. Template Variable Naming: Ensure that the template variable names in the payload (

build-image.image

) exactly match the ones defined in your

prefect.yaml

. 2. Proper Nesting: Make sure

job_variables

are properly nested inside the

parameters

field of the JSON payload. ### Documentation References - Overriding Work Pool Job Variables explains the usage of job variables in deployments and how they can be overridden. - Using the Prefect API gives a more comprehensive understanding of interacting with the Prefect API. This should ensure that your deployment correctly uses the specified image.

Martijn van Elferen

06/19/2024, 9:48 AM

@Marvin, without taking in the previous context. How can I override a static job variable (image) in a deployment for a single run?

Marvin

06/19/2024, 9:48 AM

To override a static job variable (e.g.,

image

) in a deployment for a single run in Prefect, you can leverage the

job_variables

parameter. Here’s how you can do it using both the CLI and the API. ### Using the CLI The

prefect deployment run

command allows you to use the

--job-variable

flag to override job variables for a specific run. Here’s an example to override the

image

variable:

Copy code

sh
prefect deployment run \
--id "your_deployment_id" \
--job-variable "image=<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"

### Using the API When using the API to trigger a flow run, include the desired job variable overrides in the

parameters

field of your request payload. Here’s an example payload to override the

image

variable:

Copy code

json
{
  "state": {
    "type": "SCHEDULED"
  },
  "parameters": {
    "job_variables": {
      "image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
    }
  }
}

Here’s how you can send this request using Python and the

requests

library:

Copy code

python
import requests

account_id = "your_account_id"
workspace_id = "your_workspace_id"
deployment_id = "your_deployment_id"
api_key = "YOUR_PREFECT_API_KEY"

url = f"<https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{deployment_id}/create_flow_run>"

headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

payload = {
    "state": {
        "type": "SCHEDULED"
    },
    "parameters": {
        "job_variables": {
            "image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
        }
    }
}

response = <http://requests.post|requests.post>(url, headers=headers, json=payload)

if response.status_code == 201:
    print("Flow Run created successfully!")
else:
    print(f"Failed to create Flow Run: {response.text}")

### Summary 1. CLI Example:

Copy code

sh
    prefect deployment run --id "your_deployment_id" --job-variable "image=<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"

2. API Example:

Copy code

python
    import requests

    account_id = "your_account_id"
    workspace_id = "your_workspace_id"
    deployment_id = "your_deployment_id"
    api_key = "YOUR_PREFECT_API_KEY"

    url = f"<https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{deployment_id}/create_flow_run>"

    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }

    payload = {
        "state": {
            "type": "SCHEDULED"
        },
        "parameters": {
            "job_variables": {
                "image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
            }
        }
    }

    response = <http://requests.post|requests.post>(url, headers=headers, json=payload)

    if response.status_code == 201:
        print("Flow Run created successfully!")
    else:
        print(f"Failed to create Flow Run: {response.text}")

### Resources - Overriding Work Pool Job Variables explains how to override job variables in deployments. - Using the Prefect CLI for more details on using CLI commands. - Prefect API Reference for more details on interacting with the Prefect API.

Martijn van Elferen

06/19/2024, 9:54 AM

@Marvin, I don't use the parameter value, instead I use the CLI to submit the image straightaway: prefect deployment run --id "de27fd5c-eab1-4f40-8bbc-7a6684e1ceb2" \ --job-variable image="eu.gcr.io/abc/prefect/prefect-pricing:production". Executing the above will not override the job-variable

image

in the deployment, it will only show it in the flow run job variables tab.. Why?

Marvin

06/19/2024, 9:55 AM

When you use the CLI command

prefect deployment run

with the

--job-variable

flag, the expectation is that the specified job variable will override the default set in the deployment. If it's only showing in the flow run job variables tab without actually overriding the deployment configuration, it might be due to how the templating and variable substitution work in your deployment setup. ### Common Issues and Solutions 1. Ensure Correct Variable Naming and Substitution: - Ensure that the variable name matches exactly the key used in your deployment's configuration. - Ensure there are no typos or mismatches in variable names. 2. Proper Variable Substitution: - The template system should use the provided variable if correctly substituted in the

job_variables

. Ensure that your deployment definition is correctly set up to allow variable substitution. ### Calibration Example Here’s a detailed example to verify the deployment configuration and the CLI command: #### Deployment Configuration (prefect.yaml):

Copy code

yaml
deployments:
- name: my-deployment
  version: "{{ build-image.tag }}"
  tags:
      - "{{ $my_deployment_tag }}"
      - "{{ prefect.variables.some_common_tag }}"
  description: null
  schedule: null
  entrypoint: demo_project/demo_flow.py:some_work
  parameters: {}
  work_pool:
    name: "my-k8s-work-pool"
    work_queue_name: null
    job_variables:
      image: "{{ build-image.image }}"
      cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-favorite-config }}"

#### CLI Command:

Copy code

sh
prefect deployment run \
  --id "de27fd5c-eab1-4f40-8bbc-7a6684e1ceb2" \
  --job-variable image="<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"

### Post-Execution Verification 1. Check Substitution: - After running the command, ensure that the flow run picked up the new image in the logs or the flow run details. - Check the Work Pool’s job configuration in the Prefect UI to verify the variable substitution. 2. Check Logs and Output: - Look at the logs/output of the flow run to see if the new image was indeed used. - If the variable is correctly passed but not substituted, there might be an issue in how Prefect handles this specific case. ### Troubleshooting Steps: 1. Log Variables: - Print environment variables or log them inside your flow to ensure they are being correctly passed and recognized.

Copy code

python
    @flow
    def some_work():
        import os
        print(os.environ.get("MY_NEW_ENV_VAR"))
        # Or use logging

2. Test Simpler Cases: - Try with simpler job variable overrides to ensure the mechanism works before applying it to image overrides. - This helps isolate if the issue is specific to how the image variable is handled. 3. Prefect Community and Support: - If the issue persists, consider reaching out to the Prefect community or support with specific details and logs. By following these steps and ensuring the correctness in naming and setup, you should be able to successfully override the job variable for a single run using the CLI.

Martijn van Elferen

06/19/2024, 10:01 AM

@Marvin, does that mean I don't need to declare "image" as variable in the job variables of my deployment?

Marvin

06/19/2024, 10:01 AM

No, you still need to declare the

image

variable in the job variables of your deployment. The declaration in the deployment (e.g.,

prefect.yaml

) sets up a template that can later be overridden. This template specifies where and how the variable will be used. When you then run a flow and override

image

using the CLI, Prefect knows how to substitute the provided value in place of the template declaration. Here’s a concise step-by-step clarification: ### Step-by-Step Guide 1. Declare Job Variable in Deployment Configuration: Ensure that the

image

variable is declared in your deployment configuration (

prefect.yaml

Copy code

yaml
    deployments:
    - name: my-deployment
      version: "{{ build-image.tag }}"
      tags:
          - "{{ $my_deployment_tag }}"
          - "{{ prefect.variables.some_common_tag }}"
      description: null
      schedule: null
      entrypoint: demo_project/demo_flow.py:some_work
      parameters: {}
      work_pool:
        name: "my-k8s-work-pool"
        work_queue_name: null
        job_variables:
          image: "{{ build-image.image }}"
          cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-favorite-config }}"

2. Override Job Variable for a Specific Run via CLI: Use the CLI command to override the

image

variable when scheduling a flow run:

Copy code

sh
    prefect deployment run \
      --id "de27fd5c-eab1-4f40-8bbc-7a6684e1ceb2" \
      --job-variable image="<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"

### Rationale - Declaration: Declaring

image

as a job variable in the deployment configuration sets it up as a template. This tells Prefect that this field can be dynamically populated with different values. - Override: The CLI command or API payload then uses this template to insert the provided specific value (`eu.gcr.io/abc/prefect/prefect-pricing:production`) for the

image

variable, overriding the default or previously set value for that single run. ### Summary: 1. Deployment Configuration: - Ensure

image

is declared in

job_variables

prefect.yaml

. 2. Run Specific Override: - Use the

prefect deployment run

command with

--job-variable

flag to provide a specific value for the single run. ### Verification: After executing the CLI command, verify that the override worked correctly by checking: - Flow Run Details: In the Prefect UI, check the flow run details to see if the

image

variable has been correctly overridden. - Logs/Output: Examine the logs and output of the flow run to ensure the correct image is being used. By following these steps, you can effectively override the static

image

job variable in your deployment for a specific run while ensuring that Prefect knows how to substitute the variable correctly.

Martijn van Elferen

06/20/2024, 12:47 PM

@Marvin, do you know why this happens: The core of the issue is that I don't understand the following: • Work Pool has no default image set. • Deployment X has following Job Variables set:

Copy code

{
  "image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"
}

If this deployment is run with no job variables, the image used will be this default. ⬆️ So far so good. Yet, when I select

custom run

and intent to override the

image

value using the job variable below:

Copy code

{
  "image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:development|eu.gcr.io/abc/prefect/prefect-pricing:development>"
}

Then the run is triggered using the default value (production), instead of the expected image (development).

Marvin

06/20/2024, 12:48 PM

The issue seems to be with correctly overriding the

image

variable in a custom run scenario. Let's clarify the problem and troubleshoot why the expected image (

development

) is not being used. ### Key Points: 1. Default Job Variables in Deployment: - Deployment X has the job variable for

image

set to:

Copy code

yaml
    job_variables:
      image: "<http://eu.gcr.io/abc/prefect/prefect-pricing:production|eu.gcr.io/abc/prefect/prefect-pricing:production>"

2. Expected Behavior: - When running a deployment without additional job variables, it uses the default image (

production

). - When specifying a custom

image

variable in a custom run, it should override the default. ### Potential Issues and Solutions: 1. Template Declaration and Substitution: - Ensure variable names are correctly declared and consistently used. - Verify that the deployment configuration parses and uses variables correctly. 2. Using the Correct CLI Command: - Make sure you're forming the CLI command correctly. Ensure your command explicitly states the necessary overrides. ### CLI Command Example:

Copy code

sh
prefect deployment run \
  --id "de27fd5c-eab1-4f40-8bbc-7a6684e1ceb2" \
  --job-variable image="<http://eu.gcr.io/abc/prefect/prefect-pricing:development|eu.gcr.io/abc/prefect/prefect-pricing:development>"

### Troubleshooting Steps: 1. Verify Deployment Configuration: - Ensure the deployment configuration specifies the image variable correctly. - Check the Prefect UI to verify the deployment settings. 2. Use Debug Logging: - Add logging in your flow to output the image variable being used.

Copy code

python
    import os
    from prefect import flow, task

    @task
    def print_image():
        image = os.environ.get('image', 'Not Found')
        print(f"Using image: {image}")

    @flow
    def my_flow():
        print_image()

    if __name__ == "__main__":
        my_flow()

3. Check Prefect Cloud/Server: - Ensure that the Prefect server/cloud environment correctly processes the overridden variables. - Look at the flow run details to confirm that the CLI overrides are passed and applied. ### Verification via API (Consistent Troubleshooting): To ensure consistency and verify the state of the deployment variables, you may also use the API directly for more granular control and verification. #### API Request Example:

Copy code

python
import requests

account_id = "your_account_id"
workspace_id = "your_workspace_id"
deployment_id = "de27fd5c-eab1-4f40-8bbc-7a6684e1ceb2"
api_key = "YOUR_PREFECT_API_KEY"

url = f"<https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{deployment_id}/create_flow_run>"

headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

payload = {
    "state": {
        "type": "SCHEDULED"
    },
    "parameters": {
        "job_variables": {
            "image": "<http://eu.gcr.io/abc/prefect/prefect-pricing:development|eu.gcr.io/abc/prefect/prefect-pricing:development>"
        }
    }
}

response = <http://requests.post|requests.post>(url, headers=headers, json=payload)

if response.status_code == 201:
    print("Flow Run created successfully!")
else:
    print(f"Failed to create Flow Run: {response.text}")

### Documentation Reference: For further guidance, refer to the Prefect documentation on Overriding Job Variables. ### Summary: 1. Verify Deployment Configuration: - Ensure

image

is correctly set as a job variable. 2. Use CLI/Custom Run with Override: - Ensure the overriding variable is properly passed in the CLI command or API request. 3. Debug and Verify: - Use logging in the flow and check Prefect UI or server logs/outputs to confirm variable values.

Marvin

06/20/2024, 12:48 PM

By following these steps and ensuring the correct setup and overrides, the deployment should use the overridden

image

variable for the custom run as expected.

7 Views

Open in Slack

Previous Next