https://prefect.io
Join Slack
Happy New Year everyone! When I run a GE checkpoint in GE it works, but in prefect task, I'm getting...
# ask-community
d
Happy New Year everyone! When I run a GE checkpoint in GE it works, but in prefect task, I'm getting the error:
Copy code
TypeError: 'Checkpoint' object is not subscriptable
k
What Prefect version are you on?
d
its 0.15.5
k
There was an update to this task in 0.15.11
d
Ok, so we need to update to 0.15.11
thank you
Hi @Kevin Kho we now have the latest prefect version but still getting the checkpoint error, do you know what else could be the problem? thank you
a
Hey dammy, what version of 
great_expectations
do you have installed?
d
Copy code
0.13.47
a
Cool, and are you using GE’s v2 or v3 API?
d
v3 API
a
That all checks out. Do you have a full stack trace of the error that you can share?
d
Copy code
Task 'RunGreatExpectationsValidation': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 454, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/great_expectations/checkpoints.py", line 246, in run
    for batch in ge_checkpoint["batches"]:
TypeError: 'Checkpoint' object is not subscriptable
a
That stack trace makes me think that you’re running an old version of the task, because the line that is causing the error was removed in the last update. It looks like you’re running this task on Prefect Cloud. What type of agent are you using?
d
Kubernetes
k
What image are you using for KubernetesRun?
d
Docker image
k
I’m asking if the Docker image is a different person than your Flow registration version 😅
d
😀
It's the same
Copy code
{
  "env": null,
  "type": "KubernetesRun",
  "image": "molsonlabs/prefect-snowflake:0.0.9",
  "labels": [],
  "cpu_limit": null,
  "__version__": "0.15.11",
  "cpu_request": null,
  "job_template": null,
  "memory_limit": null,
  "memory_request": null,
  "image_pull_policy": null,
  "job_template_path": "<s3://iddoc-shared-dev/prefect/k8s-templates/ge_template.yaml>",
  "image_pull_secrets": null,
  "service_account_name": null
}
k
What executor are you using?
d
Is this the same as execution engine on GE? If so, its SqlAlchemyExecutionEngine
k
No the Prefect Executor. Just checking if you are using DaskExecutor
d
oh ok, let me ask someone in our DevOps. I don't have executor specified anywhere in my flow script.
k
Oh then you should just be using the default…I’m not sure where the error is coming from. Alex would be better here
a
Do you know what the base image of 
molsonlabs/prefect-snowflake:0.0.9
is?
d
Hi @alex, so sorry I'm just responding, I was off last week. We updated the image also and the checkpoint error is gone.
We are now getting another error 😭
Copy code
Task 'RunGreatExpectationsValidation': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 467, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/great_expectations/checkpoints.py", line 245, in run
    run_id={"run_name": run_name or prefect.context.get("task_slug")},
  File "/usr/local/lib/python3.7/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 307, in usage_statistics_wrapped_method
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/checkpoint/checkpoint.py", line 390, in run
    validation_dict=validation_dict,
  File "/usr/local/lib/python3.7/site-packages/great_expectations/checkpoint/checkpoint.py", line 240, in _run_validation
    else None
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 1732, in get_validator
    expectation_suite = self.get_expectation_suite(expectation_suite_name)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 2204, in get_expectation_suite
    "expectation_suite %s not found" % expectation_suite_name
great_expectations.exceptions.exceptions.DataContextError: expectation_suite cms_ge_exp not found
It's not finding the expectation suite even though everything is in the s3 bucket and it's finding the checkpoint
a
Are you downloading your 
great_expections
folder from S3 in a earlier task in your flow?
d
Yes, we are
Copy code
s3FileName = 'cms2_ge_exp.json'
        s3CheckPoint = 'cms_ge_checkpoint.yml'
        fileReadPath = WORKING_DIR + '/'+ s3FileName
       
        CPReadPath = WORKING_DIR + '/checkpoints/' + s3CheckPoint
        <http://logger.info|logger.info>('fileReadPath: ' + fileReadPath)
       
        s3CopyData = s3Copy('iddoc-shared-dev','prefect/ge/' + s3FileName, fileReadPath)
       
        s3CopyDataCP = s3Copy('iddoc-shared-dev','prefect/ge/' + s3CheckPoint, CPReadPath)
a
By default, GE looks for expectations in the expectations folder. Can you try downloading your expectation suite into a folder named 
expectations
nested under 
WORKING_DIR
? Also, your expectation suite file is named 
cms2_ge_exp.json
, but the error that you saw looks like you’ve configured the expectation suite name to be 
cms_ge_exp
. Those names should match so that GE can correctly find the expectation suite.
d
Yes, I tried to create the expectations folder
Copy code
fileReadPath = WORKING_DIR + '/expectations/' + s3FileName
so it's downloaded and goes into that, but another error comes from there that the datasource is not found
I created another expectation suite, that's why the name is different in the code
Here's the error after using the expectations folder
I downloaded the greatexpectations.yml file into the working_dir and it's working now
Thank you so much!
19 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by