https://prefect.io logo
Title
i

Ilya Galperin

08/19/2022, 3:48 PM
When using the python
Deployment
object and specifying a remote S3 flow storage block and Kubernetes infrastructure, we are seeing strange behavior on flow execution. The deployment pushes the flow to S3 storage as expected (which is confirmed by the storage block in the Prefect Cloud UI being referenced in the deployment UI) but errs out with the following:
Flow could not be retrieved from deployment...FileNotFoundError: [Errno 2] No such file or directory: '/My/Local/Path/my_project/flow.py'
where the path is the absolute path of the machine that applied the deployment whereas the absolute path in the s3 bucket is just
<bucketname://flow.py>
. Here is the code we are using if anyone has any ideas?
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob
from prefect.filesystems import S3
from my_project.flow import entrypoint

infrastructure = KubernetesJob(namespace="prefect2")

deployment = Deployment.build_from_flow(
    flow=entrypoint,
    name="my_deployment",
    work_queue_name="default",
    storage=S3.load("default-block"),
    infrastructure=infrastructure,
)
deployment.apply()
k

Khuyen Tran

08/19/2022, 3:57 PM
Did you get this error after running an agent?
i

Ilya Galperin

08/19/2022, 3:59 PM
Hi Khuyen - this is the error on the pod/container running the actual flow, the agent successfully starts a new pod for the flow.
k

Khuyen Tran

08/19/2022, 4:12 PM
Hmm. That is a weird error. It could be that the agent picked up another deployment (not the one that uses S3). Can you double check the name of the work queue and see if your agent is running on the right work queue?
i

Ilya Galperin

08/19/2022, 4:17 PM
Double checked and the agent is also running the “default” queue specified in our Deployment object; there are also no other deployments scheduled in this environment, just kicking this one-off from the Run tab in Cloud UI and I can see it fail in real time.
This is what the deployment shows under its UI in Cloud
k

Khuyen Tran

08/19/2022, 4:19 PM
Thank you for checking. Do you mind taking the screenshot of the path of your S3 as well? Just to double check
i

Ilya Galperin

08/19/2022, 4:19 PM
Sure
This is the flow as it appears in the storage block/bucket referenced by the deployment in the root dir of the bucket.
d

David Hlavaty

08/19/2022, 4:21 PM
This is the same cause as this issue: https://github.com/PrefectHQ/prefect/issues/6469
:thank-you: 1
i

Ilya Galperin

08/19/2022, 4:23 PM
It seems like the container is actually trying to maybe pull from s3 (?) because when i remove s3fs from the flow container’s image, i get an s3fs missing/remote filesystem error
Thanks for that link David. I’m getting the same behavior it looks like (including the CLI build/apply in the same folder working)
r

Ross Teach

08/19/2022, 4:25 PM
d

David Hlavaty

08/19/2022, 4:25 PM
My workaround is to fix the entrypoint before calling `apply`:
def fix_entrypoint(entrypoint: str) -> str:
    # Workaround until <https://github.com/PrefectHQ/prefect/issues/6469> is resolved
    flow_path, flow = entrypoint.split(':')
    flow_path = Path(flow_path).relative_to(Path('.').absolute())
    return f"{flow_path}:{flow}"

d = Deployment.build_from_flow(...)
d.entrypoint = fix_entrypoint(d.entrypoint)
d.apply()
🙌 2
i

Ilya Galperin

08/19/2022, 4:29 PM
Thank you David! Can you describe what is being passed to the
entrypoint
argument is in this function?
k

Khuyen Tran

08/19/2022, 4:29 PM
Thank you for raising this issue and the workaround. I’ll talk about this to the team
i

Ilya Galperin

08/19/2022, 4:30 PM
Thank you Khuyen. Would it help if I also opened up an issue on github?
k

Khuyen Tran

08/19/2022, 4:30 PM
Yes, that would be very helpful. Thank you. I’ll link to your issue in the existing issue
:thank-you: 1
We will fix this as soon as possible. For now, I recommend that you use the CLI to create the deployment.
👍 1
i

Ilya Galperin

08/19/2022, 4:46 PM
Hi Khuyen — I opened an issue here: https://github.com/PrefectHQ/prefect/issues/6482
:gratitude-thank-you: 1
k

Khuyen Tran

08/19/2022, 6:10 PM
Thank you for the descriptive issue! We will try to fix this ASAP
👍 1
d

David Hlavaty

08/22/2022, 8:59 AM
Thank you David! Can you describe what is being passed to the
entrypoint
argument is in this function?
The original
entrypoint
as set by
Deployment.build_from_flow
. It is an absolute path, and the workaround converts it to a path relative to the working directory, which is what
build_from_flow
uses when uploading the files to your target storage.