Scott Walsh
09/11/2022, 1:00 AMCA Lee
09/11/2022, 8:27 AMis_schedule_active
property in deployment.yaml?
Error:
'deployments/func-deployment.yaml' did not conform to deployment spec: ValidationError(model='Deployment', errors=[{'loc': ('is_schedule_active',), 'msg': 'extra fields not permitted', 'type': 'value_error.extra'}])
This is using S3 block storage. If using LocalFileStorage, I believe this error does not occur.
Prefect version: 2.3.2
Removing the field: is_schedule_active: false
, the deployment can be applied successfully, but the user would need to login to the UI to manually disable the schedule. This is especially painful if triggered on each CI/CD run.Ammar Chalifah
09/11/2022, 6:23 PMPREFECT_API_URL=http//0.0.0.0:4200/api
Chris L.
09/11/2022, 6:51 PM[ "python", "-m", "prefect.engine" ]
for KubernetesJob
18:49:52.809 | INFO | prefect.agent - Submitting flow run 'faa2ebac-ee94-4384-9a4f-16ad28d6fa7e'
....
File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 171, in __init__
super().__init__(*args, **kwargs)
File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KubernetesJob
command
none is not an allowed value (type=type_error.none.not_allowed)
James Constable
09/12/2022, 2:53 AMAhmed Ezzat
09/12/2022, 4:07 AMNaila Chennit
09/12/2022, 8:41 AMEli Treuherz
09/12/2022, 9:01 AMraise SKIP("not applicable")
in order to skip a task? If I just return Cancelled(message="not applicable")
instead of returning my task’s actual data, will that have the same effect?Dennis Hinnenkamp
09/12/2022, 12:10 PMsnowflake_connector_block
for a dbt shell task, but I get the following error when I use the created `dbt_cli_profile`:
14:07:11.533 | ERROR | Task run 'trigger_dbt_cli_command-321ca940-0' - Encountered exception during execution:
Traceback (most recent call last):
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect_dbt/cli/commands.py", line 136, in trigger_dbt_cli_command
profile = dbt_cli_profile.get_profile()
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect_dbt/cli/credentials.py", line 107, in get_profile
"outputs": {self.target: self.target_configs.get_configs()},
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect_dbt/cli/configs/snowflake.py", line 78, in get_configs
configs_json = super().get_configs()
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect_dbt/cli/configs/base.py", line 53, in get_configs
return self._populate_configs_json({}, self.dict())
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect_dbt/cli/configs/base.py", line 34, in _populate_configs_json
configs_json = self._populate_configs_json(configs_json, value)
File "/Users/d.hinnenkamp/Desktop/test-prefect/.venv/lib/python3.8/site-packages/prefect_dbt/cli/configs/base.py", line 37, in _populate_configs_json
raise ValueError(
ValueError: The keyword, _is_anonymous, has already been provided in TargetConfigs; remove duplicated keywords to continue
My current code looks like this:
snowflake_connector_block = await SnowflakeConnector.load("wld-snowflake-connection")
connector = snowflake_connector_block
target_configs = SnowflakeTargetConfigs(
connector=connector
)
dbt_cli_profile = DbtCliProfile(
name="profile_name",
target="dev",
target_configs=target_configs
)
debug = await trigger_dbt_cli_command.submit(
"dbt debug",
profiles_dir='profile',
project_dir='wld-dbt-warehouse/dbt-project',
overwrite_profiles=True,
dbt_cli_profile=dbt_cli_profile,
return_state=True
)
Vadym Dytyniak
09/12/2022, 12:22 PMcustomizations=[
{
"op": "add",
"path": "/spec/imagePullSecrets",
"value": [{'name': 'dockerhub'}],
},
],
What is wrong with passed value? I still get an access denied error.Niels Prins
09/12/2022, 1:14 PMfilters
when creating a WorkQueue?
<EDIT> Yes this is correct, see threadPrasanth Kothuri
09/12/2022, 2:03 PMPrasanth Kothuri
09/12/2022, 2:03 PMschedule = Schedule(clocks=[CronClock("0 07,12,14 * * 1-6"),CronClock("0 10 * * 0")])
Prasanth Kothuri
09/12/2022, 2:05 PMKelvin Garcia
09/12/2022, 2:51 PMJosh Cowles
09/12/2022, 3:19 PMSam Garvis
09/12/2022, 5:36 PMValueError: Path /root/.prefect/storage/72ed22dddb494741872cffe99e581ee9 does not exist.
It's an incredibly vague error that is hard to debug. However, the flows all complete successfully, they just show as crashed. But if we were to transition to 2.0, we need reliable error monitoring, and we can't do that with 90% of flows crashing for no reason.Nathaniel Russell
09/12/2022, 6:42 PMKelvin Garcia
09/12/2022, 7:20 PM{
"jobId": reportId,
"s3Location": s3Location,
"validationSettings": validation_settings,
}
but when the flow fails and I need to restart a task that uses that dict as a dependency I get this error
reportId = settings["jobId"]
TypeError: 'NoneType' object is not subscriptable
why the second time I run that task the input is None
, shouldn't it cache the input for restarts?Leon Kozlowski
09/12/2022, 7:55 PMprefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/work_queues/>'
Not sure if I have my api url wrong or some other configuration incorrectjack
09/12/2022, 9:15 PMChris Gunderson
09/12/2022, 10:02 PMNiels Prins
09/13/2022, 8:07 AMAfaque Ahmad
09/13/2022, 8:08 AMAndreas Nord
09/13/2022, 8:26 AM08:12:06.349 | INFO | prefect.infrastructure.docker-container - Docker container 'practical-bear' has status 'exited'
But the log on prefect cloud is stuck on Executing of the last task of the flow.Saurabh Indoria
09/13/2022, 9:01 AMDennis Hinnenkamp
09/13/2022, 9:20 AM09:19:54.856 | INFO | prefect.agent - Submitting flow run '9e981ee3-5d70-43b8-992a-6bd5c433d24c'
09:19:55.054 | INFO | prefect.infrastructure.process - Opening process 'blond-rabbit'...
09:19:55.130 | INFO | prefect.agent - Completed submission of flow run '9e981ee3-5d70-43b8-992a-6bd5c433d24c'
09:19:57.018 | INFO | Flow run 'blond-rabbit' - Created task run 'shell_run_command-7398b6ba-0' for task 'shell_run_command'
09:19:57.019 | INFO | Flow run 'blond-rabbit' - Executing 'shell_run_command-7398b6ba-0' immediately...
09:19:57.251 | INFO | Task run 'shell_run_command-7398b6ba-0' - Soda Core 3.0.6
09:19:58.517 | INFO | Task run 'shell_run_command-7398b6ba-0' - Attempting to acquire lock 281472472552592 on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:19:58.518 | INFO | Task run 'shell_run_command-7398b6ba-0' - Lock 281472472552592 acquired on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:19:58.518 | INFO | Task run 'shell_run_command-7398b6ba-0' - Attempting to acquire lock 281472472682848 on /home/vscode/.cache/snowflake/ocsp_cache.lock
Timeout on acquiring lock 281472472682848 on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:19:58.519 | INFO | Task run 'shell_run_command-7398b6ba-0' - Attempting to release lock 281472472552592 on /home/vscode/.cache/snowflake/ocsp_cache.lock
Lock 281472472552592 released on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:20:00.123 | INFO | Task run 'shell_run_command-7398b6ba-0' - Scan summary:
1/1 check FAILED:
v_stg_transactions in we_liberate_data
row_count between 10 and 1000 [FAILED]
09:20:00.125 | INFO | Task run 'shell_run_command-7398b6ba-0' - check_value: 6209
Oops! 1 failures. 0 warnings. 0 errors. 0 pass.
09:20:00.125 | INFO | Task run 'shell_run_command-7398b6ba-0' - Sending results to Soda Cloud
09:20:00.484 | INFO | Task run 'shell_run_command-7398b6ba-0' - Soda Cloud Trace: 3791758754192003542
09:20:00.878 | INFO | Task run 'shell_run_command-7398b6ba-0' - Attempting to acquire lock 281472486055312 on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:20:00.878 | INFO | Task run 'shell_run_command-7398b6ba-0' - Lock 281472486055312 acquired on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:20:00.879 | INFO | Task run 'shell_run_command-7398b6ba-0' - Attempting to release lock 281472486055312 on /home/vscode/.cache/snowflake/ocsp_cache.lock
Lock 281472486055312 released on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:20:00.930 | ERROR | Task run 'shell_run_command-7398b6ba-0' - Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Lock 281472486055312 released on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:20:00.952 | ERROR | Task run 'shell_run_command-7398b6ba-0' - Finished in state Failed('Task run encountered an exception.')
09:20:00.952 | ERROR | Flow run 'blond-rabbit' - Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 588, in orchestrate_flow_run
result = await flow_call()
File "flows/test/test_soda_transactions.py", line 11, in test_transactions
command2 = await shell_run_command(
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 863, in get_task_call_return_value
return await future._result()
File "/usr/local/lib/python3.8/site-packages/prefect/futures.py", line 236, in _result
return final_state.result(raise_on_failure=raise_on_failure)
File "/usr/local/lib/python3.8/site-packages/prefect/orion/schemas/states.py", line 143, in result
raise data
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect_shell/commands.py", line 90, in shell_run_command
raise RuntimeError(msg)
RuntimeError: Command failed with exit code 2:
Lock 281472486055312 released on /home/vscode/.cache/snowflake/ocsp_cache.lock
09:20:00.966 | ERROR | Flow run 'blond-rabbit' - Finished in state Failed('Flow run encountered an exception.')
09:20:01.240 | INFO | prefect.infrastructure.process - Process 'blond-rabbit' exited cleanly.
Amol Shirke
09/13/2022, 9:33 AMGintautas Jankus
09/13/2022, 11:13 AMSander
09/13/2022, 11:59 AM