https://prefect.io logo
Title
d

Dennis Hinnenkamp

09/13/2022, 9:20 AM
I'm trying to use a shell_run_command to run some soda.io scans again my dbt models with prefect. The execution is correct but if the test itself fails, the tasks returns an exception and so my flow is also failing. Is there anything I can do to prevent the task from terminating in an incorrect state?
09:19:54.856 | INFO    | prefect.agent - Submitting flow run '9e981ee3-5d70-43b8-992a-6bd5c433d24c'
09:19:55.054 | INFO    | prefect.infrastructure.process - Opening process 'blond-rabbit'...
09:19:55.130 | INFO    | prefect.agent - Completed submission of flow run '9e981ee3-5d70-43b8-992a-6bd5c433d24c'
09:19:57.018 | INFO    | Flow run 'blond-rabbit' - Created task run 'shell_run_command-7398b6ba-0' for task 'shell_run_command'
09:19:57.019 | INFO    | Flow run 'blond-rabbit' - Executing 'shell_run_command-7398b6ba-0' immediately...
09:19:57.251 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Soda Core 3.0.6

09:19:58.517 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Attempting to acquire lock 281472472552592 on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:19:58.518 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Lock 281472472552592 acquired on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:19:58.518 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Attempting to acquire lock 281472472682848 on /home/vscode/.cache/snowflake/ocsp_cache.lock
Timeout on acquiring lock 281472472682848 on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:19:58.519 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Attempting to release lock 281472472552592 on /home/vscode/.cache/snowflake/ocsp_cache.lock
Lock 281472472552592 released on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:20:00.123 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Scan summary:
1/1 check FAILED: 
    v_stg_transactions in we_liberate_data
      row_count between 10 and 1000 [FAILED]

09:20:00.125 | INFO    | Task run 'shell_run_command-7398b6ba-0' -         check_value: 6209
Oops! 1 failures. 0 warnings. 0 errors. 0 pass.

09:20:00.125 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Sending results to Soda Cloud

09:20:00.484 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Soda Cloud Trace: 3791758754192003542

09:20:00.878 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Attempting to acquire lock 281472486055312 on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:20:00.878 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Lock 281472486055312 acquired on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:20:00.879 | INFO    | Task run 'shell_run_command-7398b6ba-0' - Attempting to release lock 281472486055312 on /home/vscode/.cache/snowflake/ocsp_cache.lock
Lock 281472486055312 released on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:20:00.930 | ERROR   | Task run 'shell_run_command-7398b6ba-0' - Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Lock 281472486055312 released on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:20:00.952 | ERROR   | Task run 'shell_run_command-7398b6ba-0' - Finished in state Failed('Task run encountered an exception.')
09:20:00.952 | ERROR   | Flow run 'blond-rabbit' - Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 588, in orchestrate_flow_run
    result = await flow_call()
  File "flows/test/test_soda_transactions.py", line 11, in test_transactions
    command2 = await shell_run_command(
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 863, in get_task_call_return_value
    return await future._result()
  File "/usr/local/lib/python3.8/site-packages/prefect/futures.py", line 236, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/usr/local/lib/python3.8/site-packages/prefect/orion/schemas/states.py", line 143, in result
    raise data
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
    raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Lock 281472486055312 released on /home/vscode/.cache/snowflake/ocsp_cache.lock

09:20:00.966 | ERROR   | Flow run 'blond-rabbit' - Finished in state Failed('Flow run encountered an exception.')
09:20:01.240 | INFO    | prefect.infrastructure.process - Process 'blond-rabbit' exited cleanly.
j

Jean Luciano

09/13/2022, 2:28 PM
Are you returning the result of the test in the flow? Sounds like failing the test marks it a a ‘Failed’ flow. https://docs.prefect.io/concepts/states/#final-state-determination