Sebastian Gay
03/07/2023, 7:46 AMfrom prefect.infrastructure.docker import DockerContainer
from prefect.filesystems import Azure
from prefect.deployments import Deployment
from prefect import flow
az_block = Azure.load("$AzureContainerBlock")
usecase_container = DockerContainer.load("$DockerBlock")
@flow
def abc_train():
print("abc_train_internal")
docker_deploy = Deployment.build_from_flow(
flow=abc_train,
name="abc_train",
work_queue_name="test",
version="18",
storage=az_block,
path='abc',
infrastructure=usecase_container,
infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}}
)
if __name__ == "__main__":
docker_deploy.apply()
When I deploy, the upload to azure blob storage happens as expected.
However, when I run the flow (agent running on local machine) I get the following logs:
---INFO---
Downloading flow code from storage at 'abc'
---ERROR---
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine.py", line 274, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.7/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/deployments.py", line 175, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "/usr/local/lib/python3.7/site-packages/prefect/filesystems.py", line 708, in get_directory
from_path=from_path, local_path=local_path
File "/usr/local/lib/python3.7/site-packages/prefect/filesystems.py", line 322, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 293, in get
[os.makedirs(os.path.dirname(lp), exist_ok=True) for lp in lpaths]
File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 293, in <listcomp>
[os.makedirs(os.path.dirname(lp), exist_ok=True) for lp in lpaths]
File "/usr/local/lib/python3.7/os.py", line 223, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
When going line by line through retrieve_flow_then_begin_flow_run
in an interative python session everything seems to be truly fine, until the storage_block.get_directory()
call. Interactively, deployment.path
is '<az://testing-ib>'
which is the correct container name per the storage block.
Would anyone be able to help me debug this one?Christopher Boyd
03/07/2023, 6:07 PMentrypoint=<file_name.py>:abc_train
to your deploy arguments? replace file name with the name of your deployment.py file you have listedSebastian Gay
03/07/2023, 6:15 PMbuild_from_flow
right?Christopher Boyd
03/07/2023, 7:34 PM/
for your flow?