CA Lee
06/06/2021, 11:03 AM0.14.21
I am using VS Code, dev container, and running pip install prefect[aws]
in the build step . After the container is done, I am unable to do `prefect auth login -t {token}`as it leads to the error below:
vscode ➜ /workspaces/dbt-bigquery (main ✗) $ prefect auth login -t $PREFECT_AUTH_TOKEN
Traceback (most recent call last):
File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vscode/.prefect/client/https-api.prefect.io-graphql'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/cli/auth.py", line 83, in login
success_login = client.login_to_tenant(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 654, in login_to_tenant
self._save_local_settings(settings)
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 524, in _save_local_settings
self._local_settings_path.parent.mkdir(exist_ok=True, parents=True)
File "/usr/local/lib/python3.8/pathlib.py", line 1291, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/home/vscode/.prefect/client'
Jacob Blanco
06/07/2021, 6:29 AMThomas Hoeck
06/07/2021, 7:34 AMSatheesh K
06/07/2021, 10:14 AMrun_config
while building the flow, but it seems not working.João Amorim
06/07/2021, 12:35 PMJoão Amorim
06/07/2021, 12:48 PMprefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
and after some time, other error shows:
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID B: provided a running state but associated flow run A is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
does anyone have any guesses as to what is causing this? 😅Matthew Neary
06/07/2021, 6:16 PMprefect build
without pushing the image into ECR as a test for CI. I'd like to have some verification that the containers will be build correctly before registering them, and definitely don't want to be pushing up an image every time there's a PR opened.
Thanks for any feedback.Chris Vrooman
06/07/2021, 6:18 PMct
06/07/2021, 6:57 PMCM
06/07/2021, 8:10 PMClaire Herdeman
06/07/2021, 9:25 PMFelipe Saldana
06/07/2021, 10:47 PMVolker Hilsenstein
06/08/2021, 7:14 AMDaskExecutor
can handle the distribution across nodes even if no Prefect server is running. Is this correct or am I overlooking something important here?
We don't necessarily need all the Web UI things such as the nicely formatted logs and status and the ability to trigger/run registered workflows from the web UI by clicking a button.Talha
06/08/2021, 11:49 AMThomas Weatherston
06/08/2021, 12:45 PMJoseph Loss
06/08/2021, 1:49 PMJoseph Loss
06/08/2021, 1:50 PMBanjo Obayomi
06/08/2021, 2:23 PMSean Talia
06/08/2021, 3:37 PMMarwan Sarieddine
06/08/2021, 4:13 PMFlorian K. (He/Him)
06/08/2021, 4:55 PMVarun Joshi
06/08/2021, 5:17 PMHugo Kitano
06/08/2021, 11:16 PMPeter Roelants
06/09/2021, 7:26 AMStartFlowRun
. This would require having access to a Prefect server when running my tests. I want to avoid running a Prefect Server since this will tremendously slow down and complicate my unit tests.
So I was wondering if there are ways to easily mock flow runs without needing access to a Prefect server?Ben Collier
06/09/2021, 7:51 AMDomantas
06/09/2021, 8:23 AMDaskExecutor
- after the flow run is finished, DASK workers are not freeing up used RAM resources.
There are some references in Dask documentation about memory management(https://distributed.dask.org/en/latest/memory.html) but it is hard to understand how to apply these methods into Prefect code context.
Does anybody have dealt with a Prefect Dask memory management problem and knows any solution to it?Jeremy Tee
06/09/2021, 10:03 AMprefect.context.flow_run_version
is? When I use it, I am expecting it to be based off the prefect UI flow version. However, it just shows the number 3
regardless of what version my flow iscj
06/09/2021, 1:58 PMZach Schumacher
06/09/2021, 2:09 PMAdam Lewis
06/09/2021, 2:22 PMcluster.scheduler_address
around, and I'm able to create client to the cluster by running client = Client(scheduler_address)
. I want to be sure I shut down the cluster in an ending task which always runs as I've seen the scheduler stay up sometimes when an error is thrown during the flow, but I can't find a way to do that directly from the client object. I believe I can only shut down the cluster with the original cluster object which I'm not sure how to recover. Have y'all seen this and if so, do you have any suggestions for workarounds?Adam Lewis
06/09/2021, 2:22 PMcluster.scheduler_address
around, and I'm able to create client to the cluster by running client = Client(scheduler_address)
. I want to be sure I shut down the cluster in an ending task which always runs as I've seen the scheduler stay up sometimes when an error is thrown during the flow, but I can't find a way to do that directly from the client object. I believe I can only shut down the cluster with the original cluster object which I'm not sure how to recover. Have y'all seen this and if so, do you have any suggestions for workarounds?Kevin Kho
06/09/2021, 2:26 PMclient.shutdown()
work for your use case?Adam Lewis
06/09/2021, 2:27 PM