João Amorim
06/07/2021, 12:35 PMJoão Amorim
06/07/2021, 12:48 PMprefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
and after some time, other error shows:
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID B: provided a running state but associated flow run A is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
does anyone have any guesses as to what is causing this? 😅Matthew Neary
06/07/2021, 6:16 PMprefect build
without pushing the image into ECR as a test for CI. I'd like to have some verification that the containers will be build correctly before registering them, and definitely don't want to be pushing up an image every time there's a PR opened.
Thanks for any feedback.Chris Vrooman
06/07/2021, 6:18 PMct
06/07/2021, 6:57 PMCM
06/07/2021, 8:10 PMClaire Herdeman
06/07/2021, 9:25 PMFelipe Saldana
06/07/2021, 10:47 PMVolker Hilsenstein
06/08/2021, 7:14 AMDaskExecutor
can handle the distribution across nodes even if no Prefect server is running. Is this correct or am I overlooking something important here?
We don't necessarily need all the Web UI things such as the nicely formatted logs and status and the ability to trigger/run registered workflows from the web UI by clicking a button.Talha
06/08/2021, 11:49 AMThomas Weatherston
06/08/2021, 12:45 PMJoseph Loss
06/08/2021, 1:49 PMJoseph Loss
06/08/2021, 1:50 PMBanjo Obayomi
06/08/2021, 2:23 PMSean Talia
06/08/2021, 3:37 PMMarwan Sarieddine
06/08/2021, 4:13 PMFlorian K. (He/Him)
06/08/2021, 4:55 PMVarun Joshi
06/08/2021, 5:17 PMHugo Kitano
06/08/2021, 11:16 PMPeter Roelants
06/09/2021, 7:26 AMStartFlowRun
. This would require having access to a Prefect server when running my tests. I want to avoid running a Prefect Server since this will tremendously slow down and complicate my unit tests.
So I was wondering if there are ways to easily mock flow runs without needing access to a Prefect server?Ben Collier
06/09/2021, 7:51 AMDomantas
06/09/2021, 8:23 AMDaskExecutor
- after the flow run is finished, DASK workers are not freeing up used RAM resources.
There are some references in Dask documentation about memory management(https://distributed.dask.org/en/latest/memory.html) but it is hard to understand how to apply these methods into Prefect code context.
Does anybody have dealt with a Prefect Dask memory management problem and knows any solution to it?Jeremy Tee
06/09/2021, 10:03 AMprefect.context.flow_run_version
is? When I use it, I am expecting it to be based off the prefect UI flow version. However, it just shows the number 3
regardless of what version my flow iscj
06/09/2021, 1:58 PMZach Schumacher
06/09/2021, 2:09 PMAdam Lewis
06/09/2021, 2:22 PMcluster.scheduler_address
around, and I'm able to create client to the cluster by running client = Client(scheduler_address)
. I want to be sure I shut down the cluster in an ending task which always runs as I've seen the scheduler stay up sometimes when an error is thrown during the flow, but I can't find a way to do that directly from the client object. I believe I can only shut down the cluster with the original cluster object which I'm not sure how to recover. Have y'all seen this and if so, do you have any suggestions for workarounds?Greg Roche
06/09/2021, 2:46 PMMichael Wedekindt
06/09/2021, 4:26 PMwith Flow(name="dbt_flow") as f:
task = DbtShellTask(
log_stderr=True,
log_stdout=True,
return_all=True,
stream_output=True,
profiles_dir=profile_path
)(command='dbt run --models data_hub')
out = f.run()
I got this error message back
[2021-06-09 16:06:15+0200] INFO - prefect.FlowRunner | Beginning Flow run for 'dbt_flow'
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2021-06-09 16:06:15+0200] INFO - prefect.DbtShellTask | /bin/bash: C:\Users\M7856~1.\AppData\Local\Temp\prefect-5yoqhqg_: No such file or directory
[2021-06-09 16:06:15+0200] ERROR - prefect.DbtShellTask | Command failed with exit code 127
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 127')
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2021-06-09 16:06:15+0200] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
When I try this with the common ShellTask I get the same error.
Here some background facts:
I already added bash to path and I can open a bash console
I use Visual Studio Code for developing
My Python Version is 3.9.5
My prefect Version is 0.14.21
Usually I run my dbt locally in the Ubuntu Subsystem on Windows but could run dbt in windows cmd as well
Thanks in advance for help and best regards, MichaelAndrew Hannigan
06/09/2021, 6:11 PMClientException('An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.')
Fina Silva-Santisteban
06/09/2021, 6:38 PMflow.register()
method to register flows, I also set the flows’ run config and storage along with that:
flow.run_config = (some run config)
flow.storage = (some storage config)
flow.register(project_name=(project name))
I’d like to switch to using the prefect cli’s register functionality. How can I point it to the run config and storage?