Ilya Galperin
08/19/2022, 3:48 PMDeployment
object and specifying a remote S3 flow storage block and Kubernetes infrastructure, we are seeing strange behavior on flow execution. The deployment pushes the flow to S3 storage as expected (which is confirmed by the storage block in the Prefect Cloud UI being referenced in the deployment UI) but errs out with the following: Flow could not be retrieved from deployment...FileNotFoundError: [Errno 2] No such file or directory: '/My/Local/Path/my_project/flow.py'
where the path is the absolute path of the machine that applied the deployment whereas the absolute path in the s3 bucket is just <bucketname://flow.py>
. Here is the code we are using if anyone has any ideas?
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob
from prefect.filesystems import S3
from my_project.flow import entrypoint
infrastructure = KubernetesJob(namespace="prefect2")
deployment = Deployment.build_from_flow(
flow=entrypoint,
name="my_deployment",
work_queue_name="default",
storage=S3.load("default-block"),
infrastructure=infrastructure,
)
deployment.apply()
Khuyen Tran
08/19/2022, 3:57 PMIlya Galperin
08/19/2022, 3:59 PMKhuyen Tran
08/19/2022, 4:12 PMIlya Galperin
08/19/2022, 4:17 PMKhuyen Tran
08/19/2022, 4:19 PMIlya Galperin
08/19/2022, 4:19 PMDavid Hlavaty
08/19/2022, 4:21 PMIlya Galperin
08/19/2022, 4:23 PMRoss Teach
08/19/2022, 4:25 PMDavid Hlavaty
08/19/2022, 4:25 PMdef fix_entrypoint(entrypoint: str) -> str:
# Workaround until <https://github.com/PrefectHQ/prefect/issues/6469> is resolved
flow_path, flow = entrypoint.split(':')
flow_path = Path(flow_path).relative_to(Path('.').absolute())
return f"{flow_path}:{flow}"
d = Deployment.build_from_flow(...)
d.entrypoint = fix_entrypoint(d.entrypoint)
d.apply()
Ilya Galperin
08/19/2022, 4:29 PMentrypoint
argument is in this function?Khuyen Tran
08/19/2022, 4:29 PMIlya Galperin
08/19/2022, 4:30 PMKhuyen Tran
08/19/2022, 4:30 PMIlya Galperin
08/19/2022, 4:46 PMKhuyen Tran
08/19/2022, 6:10 PMDavid Hlavaty
08/22/2022, 8:59 AMThank you David! Can you describe what is being passed to theThe originalargument is in this function?entrypoint
entrypoint
as set by Deployment.build_from_flow
. It is an absolute path, and the workaround converts it to a path relative to the working directory, which is what build_from_flow
uses when uploading the files to your target storage.