Kyle McChesney
10/18/2022, 2:29 PM@resource_manager
class LocalFileManager:
...
def setup(self) -> 'LocalFileManager':
self.temp_dir = Path(mkdtemp())
self.logger = prefect.context['logger']
<http://self.logger.info|self.logger.info>('Set up LocalFileManager @ %s', self.temp_dir)
self._lock = RLock()
self._cache = {}
return self
def cleanup(self, _) -> None:
with self._lock:
<http://self.logger.info|self.logger.info>('Cleaning up: %s', self.temp_dir)
shutil.rmtree(self.temp_dir)
I had a flow fail, the setup task is the root failure. Here are the logs
17 October 2022,06:09:30 prefect.CloudTaskRunner INFO Task 'LocalFileManager.setup': Starting task run...
17 October 2022,06:09:30 prefect.LocalFileManager.setup INFO Set up LocalFileManager @ /tmp/tmp1b3_t4xs
17 October 2022,06:09:31 prefect.CloudTaskRunner INFO Task 'LocalFileManager.setup': Finished task run for task with final state: 'Success'
17 October 2022,08:55:01 prefect.CloudTaskRunner INFO Task 'LocalFileManager.setup': Starting task run...
17 October 2022,08:55:01 prefect.CloudTaskRunner ERROR Task 'LocalFileManager.setup': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/core/resource_manager.py", line 25, in run
return mgr.setup()
AttributeError: 'NoneType' object has no attribute 'setup'
17 October 2022,08:55:01 prefect.CloudTaskRunner INFO Task 'LocalFileManager.setup': Finished task run for task with final state: 'Failed'
It seems like the task ran twice and fails for the second run. This exact flow has run multiple times without issue, and this is a common resource manager used in multiple other flows.