https://prefect.io
Join Slack
Hi all, I have a question regarding using great expectations with prefect in production environment...
# ask-community
y
Hi all, I have a question regarding using great expectations with prefect in production environment, I have an agent running on EC2 and flow code stored in S3. However, whenever I deploy and run a flow that has great_expectation's 
run_checkpoint_validation
I get the following error but it runs fine locally. Also, the s3 storage used to store flow code has the great_expectations directory in it..
a
Hey @Yousef Hosny! What type of datasource is 
landing_zone
?
y
Hi Alex, It's Athena
Also, to give more context; great_expectations is installed in the prefect agent ec2 instance, I don't know what might be the issue..
a
Could there be a difference in the version of 
great_expectations
installed locally vs. the version installed on your EC2 instance?
y
Same version, 0.15.20
a
Hmmm, the next thing I would recommend checking would be if you can run your great expectations validation on that EC2 instance via 
great_expectations
directly.
j
Not sure, looks like a path reference problem. Maybe try to pass the path to the dir as an argument (not sure which package are you using, but something like 
data_context_root_dir
)
y
I did that, I even created a data context with the path and passed it to 
data_context
, still no success..
I have to understand the mechanism of prefect agent accessing the S3 flow code storage block, is there any reference to this, I tried searching the docs but couldn't find any
t
I'm having the same issue. Was this resolved?
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by