@Marvin I have a prefect 2 flow that uses a KubernetesJob which requires a customer docker image hosted in a private repo (I would pull it using this commnad

docker pull <http://foo.azurecr.io/bar/data/pipeline:dev|foo.azurecr.io/bar/data/pipeline:dev>

). How can I specify this in the definition of the KubernetesJob?

I found some information that should help you specify a custom Docker image from a private repository in a KubernetesJob for your Prefect 2 flow. In the

KubernetesJob

definition, you can set the

image

attribute to the custom Docker image you want to use. For example:

python
from prefect.infrastructure import KubernetesJob

k8s_job = KubernetesJob(image="<http://foo.azurecr.io/bar/data/pipeline:dev|foo.azurecr.io/bar/data/pipeline:dev>")

Additionally, you'll need to configure your Kubernetes cluster to have access to the private repository. This typically involves creating a Kubernetes secret with the necessary credentials and configuring the image pull secret in your KubernetesJob. You can find more information on how to do this in the Kubernetes documentation. For more details on the

KubernetesJob

class, you can refer to the Prefect documentation and the source code. I hope this helps you with your Prefect 2 flow. If you have any more questions, feel free to ask.

@Marvin I’m also testing some flows locally and I don’t want them to be registered in the prefect cloud backend (since they are just tests) is there any way to do so?

Yes, you can run your Prefect flows locally without registering them with Prefect Cloud. To do this, you can use the

run

method on your flow object. Here's an example:

python
from prefect import Flow, task

@task
def say_hello():
    print("Hello, world!")

with Flow("my_local_flow") as flow:
    hello = say_hello()

# Run the flow locally without registering it with Prefect Cloud
flow.run()

By calling

flow.run()

directly, you're executing the flow locally without any interaction with Prefect Cloud. This is useful for testing and development purposes. If you want to prevent accidental registration of your flows with Prefect Cloud, you can set the

register

attribute of your flow to `False`:

python
flow = Flow("my_local_flow", register=False)

This way, even if you accidentally call

flow.register()

later in your code, the flow won't be registered with Prefect Cloud. I hope this helps you with your local testing. If you have any more questions, feel free to ask.

@Marvin you got this wrong, please describe how I can switch to a profile that uses the ephemeral API

Apologies for the confusion. To use the Ephemeral API for testing your flows locally without registering them in Prefect Cloud, you can create a new Prefect configuration profile. Here's how you can do it: 1. Create a new configuration file, for example,

ephemeral-config.toml

, with the following content:

toml
[cloud]
api = "<https://api.prefect.io>"
auth_token = ""
backend = "server"

[server]
host = "<http://localhost>"
port = 4200

2. Set the

PREFECT__USER_CONFIG_PATH

environment variable to point to the new configuration file:

bash
export PREFECT__USER_CONFIG_PATH=/path/to/ephemeral-config.toml

3. Now, when you run your flows, they will use the Ephemeral API and won't be registered with Prefect Cloud. Remember to switch back to your original configuration when you want to use Prefect Cloud again. You can do this by either unsetting the

PREFECT__USER_CONFIG_PATH

environment variable or setting it to the path of your original configuration file. I hope this helps you with your local testing. If you have any more questions, feel free to ask.

don't listen to marvin - we've got some work to do to make him more aware of all our concepts

Thanks for the hints Nate!