@Marvin How to make relative import work when storing Prefect flows and utils source code in S3 and running in Kubernetes?

To make relative imports work when storing Prefect flows and utility source code in S3 and running them in Kubernetes, you'll need to consider the following points: 1. Packaging: Your Prefect flows and related source code should be properly packaged (i.e., using Python packages with

__init__.py

files) so that Python can recognize and handle relative imports correctly. 2. Mounting: When running flows in Kubernetes, if your source code is stored in S3, you'll need a way to make that code available to the execution environment. This could involve downloading the code at the start of the run or using a tool to mount the S3 bucket as a volume in the Kubernetes pod. 3. Flow Design: Flows should be defined in a way that supports dynamic loading of modules when the flow starts. This might require ensuring that all import statements are compatible with the way your packages are stored and accessed on S3. 4. Version Controlled Storage: Use Prefect's storage options that support version control and package management. This ensures that your flows can pull the correct version of the code with all dependencies correctly resolved. The Prefect documentation does not directly address the specific scenario of handling relative imports with S3 and Kubernetes. However, running flows with Kubernetes and using Prefect integrations with AWS services like S3 can be referenced to guide you through setting up your flows in Kubernetes and interfacing with S3. If you encounter any issues or specific error messages, you can search for GitHub issues or raise new ones in Prefect's repositories to receive community support.

do you mean a real

.relative_import

or just local helpers? for the latter, here's an example https://github.com/zzstoatzz/prefect-monorepo/blob/main/src/demo_project/uses_local_helpers.py

Hi Nate, we just need local helpers. Our code is loaded from S3 and available in the k8s job pod at

/python-flows

folder. The current working directory for the running flow is

/

, and the flow file itself is located at

/python-flows/Sandbox/example_flow_v2.py

while the helpers are located in

/python-flows/Sandbox/helpers.py

. I’ve tried these two approaches

Traceback (most recent call last):
  File "/python-flows/Sandbox/example_flow_v2.py", line 62, in duck
    from helper import plus_one
ModuleNotFoundError: No module named 'helper'

and

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 2099, in orchestrate_task_run
    result = await call.aresult()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/python-flows/Sandbox/example_flow_v2.py", line 67, in duck
    from .helper import plus_one
ModuleNotFoundError: No module named '__prefect_loader__'

i see. easiest thing would be to install your helpers as a package via

pyproject.toml

in your dockerfile and then just clone your flow code that uses it

To migrate from Prefect agents to workers, you can refer to the official Upgrade Guide from Agents to Workers. This guide outlines the differences between agents and workers and provides steps for upgrading. Key points from the guide include: - Workers are a combination of agents and infrastructure blocks; they poll a work pool for flow runs and specify the default configuration for jobs submitted to that infrastructure. - Workers are typed, meaning they are designed to work with one specific kind of infrastructure. - Infrastructure configuration is specified on the work pool and passed to each worker that polls from that work pool, which simplifies the specification of each flow's infrastructure and runtime environment. The guide offers an overview of the enhancements that come with workers, as well as detailed instructions on how to transition from using agents to using workers. It's recommended to check out this guide to fully understand the process and implications of the migration.

Converting utils code into a package was what we did with Prefect 1, but I’m curious if there’s a more natural way of importing utils/helper code in Prefect 2. Looks like the demo repo you shared is able to use local helper code. Could you explain what’s missing in our setup? Thank you!

using workers allows you to add a shell script / arbitrary setup before runtime, which is not offered with agents this example implicitly clones the repo at runtime via

from_source

https://github.com/zzstoatzz/prefect-monorepo/blob/main/src/demo_project/uses_local_helpers.py#L10-L17 which is not possible with agents