https://prefect.io logo
Title
y

Yousef Hosny

09/15/2022, 11:10 AM
Hi all, I have a question regarding using great expectations with prefect in production environment, I have an agent running on EC2 and flow code stored in S3. However, whenever I deploy and run a flow that has great_expectation's
run_checkpoint_validation
I get the following error but it runs fine locally. Also, the s3 storage used to store flow code has the great_expectations directory in it..
a

alex

09/15/2022, 1:26 PM
Hey @Yousef Hosny! What type of datasource is
landing_zone
?
y

Yousef Hosny

09/15/2022, 2:17 PM
Hi Alex, It's Athena
Also, to give more context; great_expectations is installed in the prefect agent ec2 instance, I don't know what might be the issue..
a

alex

09/15/2022, 3:06 PM
Could there be a difference in the version of
great_expectations
installed locally vs. the version installed on your EC2 instance?
y

Yousef Hosny

09/15/2022, 5:05 PM
Same version, 0.15.20
a

alex

09/15/2022, 8:02 PM
Hmmm, the next thing I would recommend checking would be if you can run your great expectations validation on that EC2 instance via
great_expectations
directly.
j

Joao Moniz

09/16/2022, 11:00 AM
Not sure, looks like a path reference problem. Maybe try to pass the path to the dir as an argument (not sure which package are you using, but something like
data_context_root_dir
)
y

Yousef Hosny

09/16/2022, 11:53 AM
I did that, I even created a data context with the path and passed it to
data_context
, still no success..
I have to understand the mechanism of prefect agent accessing the S3 flow code storage block, is there any reference to this, I tried searching the docs but couldn't find any
t

Taylor Babin

09/30/2022, 8:07 PM
I'm having the same issue. Was this resolved?