John Ramirez
03/06/2020, 5:42 PMclient.restart
in the workflow because the workflow will fail. Any thoughts or ideas on how to clear memoryChris White
03/06/2020, 6:13 PMclient.restart
in the on_exit
callback of your Flow’s execution environment: https://docs.prefect.io/cloud/execution/overview.html#environmentsJohn Ramirez
03/06/2020, 6:19 PMChris White
03/06/2020, 6:21 PMJohn Ramirez
03/06/2020, 6:24 PMChris White
03/06/2020, 6:26 PMJohn Ramirez
03/06/2020, 6:27 PMChris White
03/06/2020, 7:01 PMJohn Ramirez
03/06/2020, 7:03 PMflow.register()
the on_start
or on_exit
function also runsChris White
03/06/2020, 7:31 PMJohn Ramirez
03/06/2020, 7:35 PMChris White
03/06/2020, 7:35 PMJohn Ramirez
03/06/2020, 7:38 PMTraceback (most recent call last):
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/comm/core.py", line 218, in connect
_raise(error)
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/comm/core.py", line 203, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to '<tcp://dask-scheduler.default.svc.cluster.local:8786>' after 10 s: connect() didn't finish in time
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 92, in <module>
on_start=clear_memory()
File "main.py", line 79, in clear_memory
client = Client('dask-scheduler.default.svc.cluster.local:8786')
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/client.py", line 723, in __init__
self.start(timeout=timeout)
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/client.py", line 896, in start
sync(self.loop, self._start, **kwargs)
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync
raise exc.with_traceback(tb)
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/utils.py", line 332, in f
result[0] = yield future
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/client.py", line 991, in _start
await self._ensure_connected(timeout=timeout)
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/client.py", line 1048, in _ensure_connected
connection_args=self.connection_args,
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/comm/core.py", line 227, in connect
_raise(error)
File "/Users/johnramirez/.local/share/virtualenvs/trader-ranking-Oomc_S11/lib/python3.7/site-packages/distributed/comm/core.py", line 203, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to '<tcp://dask-scheduler.default.svc.cluster.local:8786>' after 10 s: Timed out trying to connect to '<tcp://dask-scheduler.default.svc.cluster.local:8786>' after 10 s: connect() didn't finish in time
def clear_memory():
client = Client('dask-scheduler.default.svc.cluster.local:8786')
client.restart()
ENV = RemoteEnvironment(
executor="prefect.engine.executors.DaskExecutor",
executor_kwargs={
"address": "<tcp://dask-scheduler.default.svc.cluster.local:8786>"
},
on_start=clear_memory()
)
Chris White
03/06/2020, 7:40 PMclear_memory
function in this code; you should instead be providing that function to `on_start`:
ENV = RemoteEnvironment(
executor="prefect.engine.executors.DaskExecutor",
executor_kwargs={
"address": "<tcp://dask-scheduler.default.svc.cluster.local:8786>"
},
on_start=clear_memory # <--- the change I made
)
John Ramirez
03/06/2020, 7:57 PMChris White
03/06/2020, 7:57 PM