https://prefect.io
Join Slack
question: For using the default infastructure 'Process' how do i add environment variables to that i...
# ask-community
t
question: For using the default infastructure 'Process' how do i add environment variables to that in deployment?
✅ 1
k
You can use the Process block for that
t
oh thank you
how do you do that as a python object
I should phrase that better if i'm creating a deployment from a python object how do i define the process block
k
You can start with defining the infrastructure such as this:
Copy code
from prefect.infrastructure import Process

inf = Process(
    namespace="dev",
    env={"SOME_IMPORTANT_CONFIG": "true"},
)
inf.save("my_inf")
Then load it when building a deployment:
Copy code
from my_project.flows import my_flow
from prefect.deployments import Deployment
from prefect.infrastructure.process import Process

infrastructure = Process.load("dev") # load a pre-defined block

deployment = Deployment.build_from_flow(
    flow=my_flow,
    name="test",
    work_queue_name="dev",
    infrastructure=infrastructure
)

deployment.apply()
🙌 1
t
oh awesome thank you
k
Since the default infrastructure is Process, you can also just use 
infra_overrides
to override the environment variables:
Copy code
from my_project.flows import my_flow
from prefect.deployments import Deployment

deployment = Deployment.build_from_flow(
    flow=my_flow,
    name="test",
    work_queue_name="dev",
    infra_overrides=["env.SOME_IMPORTANT_CONFIG=true"],
)

deployment.apply()
z
Note that 
env
is a dictionary so. 
env={"SOME_IMPORTANT_CONFIG": "true"}
gratitude thank you 1
t
i tried that but it wasn't working for me
i did
Copy code
deployment_hello = Deployment.build_from_flow(
    flow=hello_flow,
    name="helloworld-deployment",
    work_queue_name=name,
    schedule={},
    storage=storage,
infra_override={"env":{"VARIABLE":"value"}}
)
and the flow kept saying it didn't have the credentials
i should also mention that it's a flow trying to run great expectations
k
Does it work when you create a Process block with a custom environment variable?
t
No. I just tried it. The flow works locally when i define the environment variable in the flow
so when i do the following into the actual flow script and run it, it's fine
Copy code
os.environ["VARIABLE"] = "value"
is there somewhere else i should be defining my environment variables like in the flow script do i need to also call the environment variable?
z
Does this make sense?
Copy code
>>> Process(command=["python", "-c", "import os; print(os.environ['TEST'])"], env={"TEST": "HI!"}).run()
HI!
t
yea it worked fine. in terms of the command
how do i get that into my deployment/flow
Like the process seems to recognize the envionment variable but when i deploy with infrastructure it's not picking up that environment variable
k
I get the environment variables when running a deployment created using the method above:
Copy code
from prefect import flow, task
import os

@task
def task_1():
    password = os.environ.get('PASSWORD')
    print(password)
    return password

@flow
def flow_1():
    a = task_1()
Output:
Copy code
abc
t
my deployment is just like above
Copy code
deployment_hello = Deployment.build_from_flow(
    flow=hello_flow,
    name="helloworld-deployment",
    work_queue_name=name,
    schedule={},
    storage=storage,
infrastructure=infrastructure
)
I tested out the flow by getting the os.environ.get('VARIABLE') and it returned my password. But it's still not running great expectations. its saying the credentials arent there.
which like i said. when i run the flow locally with
Copy code
os.environ["VARIABLE"] = "value"
great expectations runs fine
Now in my flow I have this
Copy code
from prefect_great_expectations import run_checkpoint_validation

@flow
def ge_node_flow():
   ps = os.environ.get('PASS')
   result = run_checkpoint_validation(checkpoint_name = checkpoint_name,raise_on_validation_failure=False,runtime_environment={"PASS":f"{ps}"})
and again it worked locally just fine. But deployment isnt
I'm using s3 as a storage block. Is that okay? Should i be using something else?
It seems like it's an issue with great expectations not working with deployment @Khuyen Tran is there anyone who can help explain how to get great expectations to work with a prefect deployment
k
We could schedule a short call so go over this if you would like @Taylor Babin
t
That would be good thank you. I'm free for the next 2hrs
k
cool. Are you available now?
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by