@Marvin how can I use visualize() in prefect 2 and in my notebook? Running

my_flow.visualize()

just returns

<coroutine object visualize_flow at 0x16962f0c0>

. Using

async/await

does not change anything either

In Prefect 2, you can visualize your flow before running it using the

.visualize()

method. This method attempts to locally produce an image of the flow's schematic diagram without running the flow's code. Here is an excerpt from the Prefect release notes:

Until now, the only way to produce a visual schematic of a flow has been to run it and view the corresponding flow run page in the Prefect UI. Some flows, though, are time consuming or expensive to run. Now, you can get a quick sense of the structure of your flow using the

.visualize()

method. Calling this method will attempt to locally produce an image of the flow's schematic diagram without running the flow's code.

If you're getting a coroutine object, it might be due to the asynchronous nature of the function. You might need to ensure that your Jupyter notebook is set up to handle async functions. If you're still having trouble, could you please provide more details about your setup and the exact code you're running?

@Marvin I have

@flow()
def my_flow:
    ...........

async def visualize_flow():
    await my_flow.visualize()

visualize_flow()

The issue here is that you're trying to run an asynchronous function in a synchronous context. The

visualize_flow()

function is asynchronous (it has the

async

keyword), but you're trying to run it like a regular function. In Python, you can't directly call an async function from a synchronous context. You have to run it inside an event loop. In the case of Jupyter notebook, you can use the

IPython

library to run async functions. Here's how you can modify your code to run the async function in Jupyter notebook:

import IPython

@flow()
def my_flow():
    ...........

async def visualize_flow():
    await my_flow.visualize()

# Run the async function
IPython.get_ipython().run_until_complete(visualize_flow())

This will run the

visualize_flow()

function in the event loop until it completes, and should display the visualization of your flow.

@Marvin how do I pass environment variables to a deployment from python sdk and also from prefect.yaml file? Looking for both solutions

To pass environment variables to a deployment from the Python SDK, you can use the

env

parameter in the

run_config

of your flow. Here's an example:

from prefect import Flow
from prefect.run_configs import DockerRun

with Flow("my_flow") as flow:
    ...

flow.run_config = DockerRun(env={"MY_ENV_VAR": "MY_VALUE"})

In this example,

MY_ENV_VAR

is the name of the environment variable, and

MY_VALUE

is its value. This environment variable will be available in the Docker container when the flow runs. To set environment variables from the

prefect.yaml

file, you can use the

env

section. Here's an example:

yaml
# prefect.yaml
env:
  MY_ENV_VAR: MY_VALUE

In this example,

MY_ENV_VAR

is the name of the environment variable, and

MY_VALUE

is its value. This environment variable will be available in the Prefect server. Please note that these examples are for Docker and may not apply to other deployment methods. For more specific information, please refer to the Prefect documentation or provide more details about your setup.

@Marvin I am looking for kubernetes not docker

To pass environment variables to a Kubernetes deployment from the Python SDK, you can use the

env

parameter in the

KubernetesRun

configuration of your flow. Here's an example:

from prefect import Flow
from prefect.run_configs import KubernetesRun

with Flow("my_flow") as flow:
    ...

flow.run_config = KubernetesRun(env={"MY_ENV_VAR": "MY_VALUE"})

In this example,

MY_ENV_VAR

is the name of the environment variable, and

MY_VALUE

is its value. This environment variable will be available in the Kubernetes Job when the flow runs. To set environment variables from the Prefect UI, you can add them when creating a new Kubernetes work pool: 1. Click on the Work Pools tab on the left sidebar 2. Click the + button at the top of the page 3. Select Kubernetes as the work pool type 4. Click Next to configure the work pool settings 5. Add environment variables to set when starting a flow run For example, to specify Python packages to install at runtime, you can use

{"EXTRA_PIP_PACKAGES":"my_package"}

. For example,

{"EXTRA_PIP_PACKAGES":"pandas==1.2.3"}

will install pandas version 1.2.3. Please note that these examples are for Kubernetes and may not apply to other deployment methods. For more specific information, please refer to the Prefect documentation or provide more details about your setup.

@Marvin talking about yaml file. are env variables defined under a specific section?

In Prefect, environment variables can be set in the

config.toml

file, not in the

prefect.yaml

file. The

prefect.yaml

file is used for defining Prefect's infrastructure, such as the Prefect server, UI, and database, but not for setting environment variables. Here's an example of how to set environment variables in the

config.toml

file:

toml
[cloud]
api = "<https://api.prefect.io>"
auth_token = "YOUR_AUTH_TOKEN"

In this example,

api

and

auth_token

are environment variables. You can replace

"<https://api.prefect.io>"

and

"YOUR_AUTH_TOKEN"

with your actual values. You can also set environment variables using the

prefect config set

command. Here's an example:

bash
prefect config set cloud.api "<https://api.prefect.io>"
prefect config set cloud.auth_token "YOUR_AUTH_TOKEN"

In this example,

cloud.api

and

cloud.auth_token

are environment variables. You can replace

"<https://api.prefect.io>"

and

"YOUR_AUTH_TOKEN"

with your actual values. Please note that these examples are for Prefect Cloud and may not apply to other deployment methods. For more specific information, please refer to the Prefect documentation or provide more details about your setup.

@Marvin so even though image name, pull policy, and build requirements are defined under prefect.yaml, environment variables which are needed to be passed to run the container are not defined under prefect.yaml?

Yes, that's correct. The

prefect.yaml

file is used for defining Prefect's infrastructure, such as the Prefect server, UI, and database. It includes settings like the image name, pull policy, and build requirements for the Docker images used by these components. Environment variables that need to be passed to the running containers (like your flows) are typically defined in the

run_config

of your flow in your Python code, or in the

config.toml

file, not in the

prefect.yaml

file. Here's an example of how to set environment variables in the

run_config

of your flow:

from prefect import Flow
from prefect.run_configs import KubernetesRun

with Flow("my_flow") as flow:
    ...

flow.run_config = KubernetesRun(env={"MY_ENV_VAR": "MY_VALUE"})

In this example,

MY_ENV_VAR

is the name of the environment variable, and

MY_VALUE

is its value. This environment variable will be available in the Kubernetes Job when the flow runs. Please note that these examples are for Kubernetes and may not apply to other deployment methods. For more specific information, please refer to the Prefect documentation or provide more details about your setup.

@Marvin what about infra_overrides?

The

infra_overrides

is a parameter that you can use to override the infrastructure settings specified in the

prefect.yaml

file when deploying your flow. This can be useful if you want to change the infrastructure settings for a specific deployment without modifying the

prefect.yaml

file. Here's an example of how to use

infra_overrides

in your Python code:

from prefect import Flow
from prefect.run_configs import KubernetesRun

with Flow("my_flow") as flow:
    ...

flow.run_config = KubernetesRun(
    env={"MY_ENV_VAR": "MY_VALUE"},
    infra_overrides={
        "image": "my_custom_image",
        "imagePullPolicy": "Always",
    }
)

In this example,

image

and

imagePullPolicy

are infrastructure settings that are being overridden for this specific deployment. The

env

parameter is used to set environment variables for the Kubernetes Job when the flow runs. Please note that these examples are for Kubernetes and may not apply to other deployment methods. For more specific information, please refer to the Prefect documentation or provide more details about your setup.