Tarek04/25/2023, 7:49 AM
). How can I specify this in the definition of the KubernetesJob?
docker pull <http://foo.azurecr.io/bar/data/pipeline:dev|foo.azurecr.io/bar/data/pipeline:dev>
Marvin04/25/2023, 7:50 AM
definition, you can set the
attribute to the custom Docker image you want to use. For example:
Additionally, you'll need to configure your Kubernetes cluster to have access to the private repository. This typically involves creating a Kubernetes secret with the necessary credentials and configuring the image pull secret in your KubernetesJob. You can find more information on how to do this in the Kubernetes documentation. For more details on the
python from prefect.infrastructure import KubernetesJob k8s_job = KubernetesJob(image="<http://foo.azurecr.io/bar/data/pipeline:dev|foo.azurecr.io/bar/data/pipeline:dev>")
class, you can refer to the Prefect documentation and the source code. I hope this helps you with your Prefect 2 flow. If you have any more questions, feel free to ask.
Tarek04/25/2023, 2:30 PM
Marvin04/25/2023, 2:30 PM
method on your flow object. Here's an example:
python from prefect import Flow, task @task def say_hello(): print("Hello, world!") with Flow("my_local_flow") as flow: hello = say_hello() # Run the flow locally without registering it with Prefect Cloud flow.run()
directly, you're executing the flow locally without any interaction with Prefect Cloud. This is useful for testing and development purposes. If you want to prevent accidental registration of your flows with Prefect Cloud, you can set the
attribute of your flow to `False`:
This way, even if you accidentally call
python flow = Flow("my_local_flow", register=False)
later in your code, the flow won't be registered with Prefect Cloud. I hope this helps you with your local testing. If you have any more questions, feel free to ask.
Nate04/25/2023, 4:47 PM
to create an ephemeral prefect environemnt https://github.com/PrefectHQ/prefect/blob/d4168979fed4a190bad971fe81fbbe8436ee80c0/src/prefect/testing/utilities.py#L148
Marvin04/25/2023, 4:54 PM
, with the following content:
2. Set the
toml [cloud] api = "<https://api.prefect.io>" auth_token = "" backend = "server" [server] host = "<http://localhost>" port = 4200
environment variable to point to the new configuration file:
3. Now, when you run your flows, they will use the Ephemeral API and won't be registered with Prefect Cloud. Remember to switch back to your original configuration when you want to use Prefect Cloud again. You can do this by either unsetting the
bash export PREFECT__USER_CONFIG_PATH=/path/to/ephemeral-config.toml
environment variable or setting it to the path of your original configuration file. I hope this helps you with your local testing. If you have any more questions, feel free to ask.
Nate04/25/2023, 4:55 PM
Tarek04/26/2023, 7:37 AM