https://prefect.io logo
Title
t

Tarek

04/25/2023, 7:49 AM
@Marvin I have a prefect 2 flow that uses a KubernetesJob which requires a customer docker image hosted in a private repo (I would pull it using this commnad
docker pull <http://foo.azurecr.io/bar/data/pipeline:dev|foo.azurecr.io/bar/data/pipeline:dev>
). How can I specify this in the definition of the KubernetesJob?
m

Marvin

04/25/2023, 7:50 AM
I found some information that should help you specify a custom Docker image from a private repository in a KubernetesJob for your Prefect 2 flow. In the
KubernetesJob
definition, you can set the
image
attribute to the custom Docker image you want to use. For example:
python
from prefect.infrastructure import KubernetesJob

k8s_job = KubernetesJob(image="<http://foo.azurecr.io/bar/data/pipeline:dev|foo.azurecr.io/bar/data/pipeline:dev>")
Additionally, you'll need to configure your Kubernetes cluster to have access to the private repository. This typically involves creating a Kubernetes secret with the necessary credentials and configuring the image pull secret in your KubernetesJob. You can find more information on how to do this in the Kubernetes documentation. For more details on the
KubernetesJob
class, you can refer to the Prefect documentation and the source code. I hope this helps you with your Prefect 2 flow. If you have any more questions, feel free to ask.
1
t

Tarek

04/25/2023, 2:30 PM
@Marvin I’m also testing some flows locally and I don’t want them to be registered in the prefect cloud backend (since they are just tests) is there any way to do so?
m

Marvin

04/25/2023, 2:30 PM
Yes, you can run your Prefect flows locally without registering them with Prefect Cloud. To do this, you can use the
run
method on your flow object. Here's an example:
python
from prefect import Flow, task

@task
def say_hello():
    print("Hello, world!")

with Flow("my_local_flow") as flow:
    hello = say_hello()

# Run the flow locally without registering it with Prefect Cloud
flow.run()
By calling
flow.run()
directly, you're executing the flow locally without any interaction with Prefect Cloud. This is useful for testing and development purposes. If you want to prevent accidental registration of your flows with Prefect Cloud, you can set the
register
attribute of your flow to `False`:
python
flow = Flow("my_local_flow", register=False)
This way, even if you accidentally call
flow.register()
later in your code, the flow won't be registered with Prefect Cloud. I hope this helps you with your local testing. If you have any more questions, feel free to ask.
n

Nate

04/25/2023, 4:47 PM
@Marvin you got this wrong, please describe how I can switch to a profile that uses the ephemeral API
@Tarek if you create a new profile with no configuration, you can use that and the ephemeral API will be used or instead, which is what we do for testing, you can use this
prefect_test_harness
to create an ephemeral prefect environemnt https://github.com/PrefectHQ/prefect/blob/d4168979fed4a190bad971fe81fbbe8436ee80c0/src/prefect/testing/utilities.py#L148
m

Marvin

04/25/2023, 4:54 PM
Apologies for the confusion. To use the Ephemeral API for testing your flows locally without registering them in Prefect Cloud, you can create a new Prefect configuration profile. Here's how you can do it: 1. Create a new configuration file, for example,
ephemeral-config.toml
, with the following content:
toml
[cloud]
api = "<https://api.prefect.io>"
auth_token = ""
backend = "server"

[server]
host = "<http://localhost>"
port = 4200
2. Set the
PREFECT__USER_CONFIG_PATH
environment variable to point to the new configuration file:
bash
export PREFECT__USER_CONFIG_PATH=/path/to/ephemeral-config.toml
3. Now, when you run your flows, they will use the Ephemeral API and won't be registered with Prefect Cloud. Remember to switch back to your original configuration when you want to use Prefect Cloud again. You can do this by either unsetting the
PREFECT__USER_CONFIG_PATH
environment variable or setting it to the path of your original configuration file. I hope this helps you with your local testing. If you have any more questions, feel free to ask.
1
n

Nate

04/25/2023, 4:55 PM
don't listen to marvin - we've got some work to do to make him more aware of all our concepts
t

Tarek

04/26/2023, 7:37 AM
Thanks for the hints Nate!