https://prefect.io logo
Powered by
Title
t

Taylor Babin

09/30/2022, 3:17 PM
question: For using the default infastructure 'Process' how do i add environment variables to that in deployment?
✅ 1
k

Khuyen Tran

09/30/2022, 3:18 PM
You can use the Process block for that
t

Taylor Babin

09/30/2022, 3:19 PM
oh thank you
how do you do that as a python object
I should phrase that better if i'm creating a deployment from a python object how do i define the process block
k

Khuyen Tran

09/30/2022, 3:30 PM
You can start with defining the infrastructure such as this: 
from prefect.infrastructure import Process

inf = Process(
    namespace="dev",
    env={"SOME_IMPORTANT_CONFIG": "true"},
)
inf.save("my_inf")
Then load it when building a deployment: 
from my_project.flows import my_flow
from prefect.deployments import Deployment
from prefect.infrastructure.process import Process

infrastructure = Process.load("dev") # load a pre-defined block

deployment = Deployment.build_from_flow(
    flow=my_flow,
    name="test",
    work_queue_name="dev",
    infrastructure=infrastructure
)

deployment.apply()
🙌 1
t

Taylor Babin

09/30/2022, 3:31 PM
oh awesome thank you
k

Khuyen Tran

09/30/2022, 3:33 PM
Since the default infrastructure is Process, you can also just use 
infra_overrides
to override the environment variables: 
from my_project.flows import my_flow
from prefect.deployments import Deployment

deployment = Deployment.build_from_flow(
    flow=my_flow,
    name="test",
    work_queue_name="dev",
    infra_overrides=["env.SOME_IMPORTANT_CONFIG=true"],
)

deployment.apply()
z

Zanie

09/30/2022, 3:34 PM
Note that 
env
is a dictionary so. 
env={"SOME_IMPORTANT_CONFIG": "true"}
:gratitude-thank-you: 1
t

Taylor Babin

09/30/2022, 3:34 PM
i tried that but it wasn't working for me
i did 
deployment_hello = Deployment.build_from_flow(
    flow=hello_flow,
    name="helloworld-deployment",
    work_queue_name=name,
    schedule={},
    storage=storage,
infra_override={"env":{"VARIABLE":"value"}}
)
and the flow kept saying it didn't have the credentials
i should also mention that it's a flow trying to run great expectations
k

Khuyen Tran

09/30/2022, 3:44 PM
Does it work when you create a Process block with a custom environment variable?
t

Taylor Babin

09/30/2022, 3:44 PM
No. I just tried it. The flow works locally when i define the environment variable in the flow
so when i do the following into the actual flow script and run it, it's fine 
os.environ["VARIABLE"] = "value"
is there somewhere else i should be defining my environment variables like in the flow script do i need to also call the environment variable?
z

Zanie

09/30/2022, 3:52 PM
Does this make sense? 
>>> Process(command=["python", "-c", "import os; print(os.environ['TEST'])"], env={"TEST": "HI!"}).run()
HI!
t

Taylor Babin

09/30/2022, 3:55 PM
yea it worked fine. in terms of the command
how do i get that into my deployment/flow
Like the process seems to recognize the envionment variable but when i deploy with infrastructure it's not picking up that environment variable
k

Khuyen Tran

09/30/2022, 4:07 PM
I get the environment variables when running a deployment created using the method above: 
from prefect import flow, task
import os

@task
def task_1():
    password = os.environ.get('PASSWORD')
    print(password)
    return password

@flow
def flow_1():
    a = task_1()
Output: 
abc
t

Taylor Babin

09/30/2022, 4:14 PM
my deployment is just like above 
deployment_hello = Deployment.build_from_flow(
    flow=hello_flow,
    name="helloworld-deployment",
    work_queue_name=name,
    schedule={},
    storage=storage,
infrastructure=infrastructure
)
I tested out the flow by getting the os.environ.get('VARIABLE') and it returned my password. But it's still not running great expectations. its saying the credentials arent there.
which like i said. when i run the flow locally with 
os.environ["VARIABLE"] = "value"
great expectations runs fine
Now in my flow I have this 
from prefect_great_expectations import run_checkpoint_validation

@flow
def ge_node_flow():
   ps = os.environ.get('PASS')
   result = run_checkpoint_validation(checkpoint_name = checkpoint_name,raise_on_validation_failure=False,runtime_environment={"PASS":f"{ps}"})
and again it worked locally just fine. But deployment isnt
I'm using s3 as a storage block. Is that okay? Should i be using something else?
It seems like it's an issue with great expectations not working with deployment @Khuyen Tran is there anyone who can help explain how to get great expectations to work with a prefect deployment
k

Khuyen Tran

09/30/2022, 8:08 PM
We could schedule a short call so go over this if you would like @Taylor Babin
t

Taylor Babin

09/30/2022, 8:10 PM
That would be good thank you. I'm free for the next 2hrs
k

Khuyen Tran

09/30/2022, 8:11 PM
cool. Are you available now?
3 Views
#prefect-communityJoin Slack
Powered by