Taylor Babin
09/30/2022, 3:17 PMKhuyen Tran
09/30/2022, 3:18 PMTaylor Babin
09/30/2022, 3:19 PMKhuyen Tran
09/30/2022, 3:30 PMfrom prefect.infrastructure import Process
inf = Process(
namespace="dev",
env={"SOME_IMPORTANT_CONFIG": "true"},
)
inf.save("my_inf")
Then load it when building a deployment:
from my_project.flows import my_flow
from prefect.deployments import Deployment
from prefect.infrastructure.process import Process
infrastructure = Process.load("dev") # load a pre-defined block
deployment = Deployment.build_from_flow(
flow=my_flow,
name="test",
work_queue_name="dev",
infrastructure=infrastructure
)
deployment.apply()
Taylor Babin
09/30/2022, 3:31 PMKhuyen Tran
09/30/2022, 3:33 PMinfra_overrides
to override the environment variables:
from my_project.flows import my_flow
from prefect.deployments import Deployment
deployment = Deployment.build_from_flow(
flow=my_flow,
name="test",
work_queue_name="dev",
infra_overrides=["env.SOME_IMPORTANT_CONFIG=true"],
)
deployment.apply()
Zanie
09/30/2022, 3:34 PMenv
is a dictionary so. env={"SOME_IMPORTANT_CONFIG": "true"}
Taylor Babin
09/30/2022, 3:34 PMdeployment_hello = Deployment.build_from_flow(
flow=hello_flow,
name="helloworld-deployment",
work_queue_name=name,
schedule={},
storage=storage,
infra_override={"env":{"VARIABLE":"value"}}
)
and the flow kept saying it didn't have the credentialsKhuyen Tran
09/30/2022, 3:44 PMTaylor Babin
09/30/2022, 3:44 PMos.environ["VARIABLE"] = "value"
Zanie
09/30/2022, 3:52 PM>>> Process(command=["python", "-c", "import os; print(os.environ['TEST'])"], env={"TEST": "HI!"}).run()
HI!
Taylor Babin
09/30/2022, 3:55 PMKhuyen Tran
09/30/2022, 4:07 PMfrom prefect import flow, task
import os
@task
def task_1():
password = os.environ.get('PASSWORD')
print(password)
return password
@flow
def flow_1():
a = task_1()
Output:
abc
Taylor Babin
09/30/2022, 4:14 PMdeployment_hello = Deployment.build_from_flow(
flow=hello_flow,
name="helloworld-deployment",
work_queue_name=name,
schedule={},
storage=storage,
infrastructure=infrastructure
)
I tested out the flow by getting the os.environ.get('VARIABLE') and it returned my password. But it's still not running great expectations. its saying the credentials arent there.os.environ["VARIABLE"] = "value"
great expectations runs finefrom prefect_great_expectations import run_checkpoint_validation
@flow
def ge_node_flow():
ps = os.environ.get('PASS')
result = run_checkpoint_validation(checkpoint_name = checkpoint_name,raise_on_validation_failure=False,runtime_environment={"PASS":f"{ps}"})
and again it worked locally just fine. But deployment isntKhuyen Tran
09/30/2022, 8:08 PMTaylor Babin
09/30/2022, 8:10 PMKhuyen Tran
09/30/2022, 8:11 PM