Riley Hun

    1 year ago
    Hello all - I feel like I'm super close to getting my flow productionized and integrated with Dask Gateway. Unfortunately, came across a
    (see reply for full stacktrace of error). Tried to look through the thread to see if others have encountered a similar error, but no luck. Any insight on this?
    Error from the logs:
    Unexpected error: PermissionError(13, 'Permission denied')
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/opt/conda/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 500, in get_flow_run_state
        {e: state for e, state in upstream_states.items()}
      File "/opt/conda/lib/python3.7/site-packages/prefect/engine/executors/dask.py", line 451, in wait
        return self.client.gather(futures)
      File "/opt/conda/lib/python3.7/site-packages/distributed/client.py", line 1992, in gather
      File "/opt/conda/lib/python3.7/site-packages/distributed/client.py", line 833, in sync
        self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
      File "/opt/conda/lib/python3.7/site-packages/distributed/utils.py", line 340, in sync
        raise exc.with_traceback(tb)
      File "/opt/conda/lib/python3.7/site-packages/distributed/utils.py", line 324, in f
        result[0] = yield future
      File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 762, in run
        value = future.result()
      File "/opt/conda/lib/python3.7/site-packages/distributed/client.py", line 1851, in _gather
        raise exception.with_traceback(traceback)
      File "/opt/conda/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 725, in run_task
      File "/opt/conda/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 52, in __init__
        self.client = Client()
      File "/opt/conda/lib/python3.7/site-packages/prefect/client/client.py", line 109, in __init__
        settings = self._load_local_settings()
      File "/opt/conda/lib/python3.7/site-packages/prefect/client/client.py", line 473, in _load_local_settings
        if self._local_settings_path.exists():
      File "/opt/conda/lib/python3.7/pathlib.py", line 1361, in exists
      File "/opt/conda/lib/python3.7/pathlib.py", line 1183, in stat
        return self._accessor.stat(self)
    PermissionError: [Errno 13] Permission denied: '/root/.prefect/client/http-'
    I should also add that when the flow runs, it does create the ephemeral Dask Cluster via Dask Gateway. So that's one thing down. 🙂
    Okay nevermind - got it working by modifying the dockerfile used for the Dask workers. 🙂