https://prefect.io logo
#data-tricks-and-tips
Title
# data-tricks-and-tips
j

John Kang

07/26/2022, 3:40 PM
Hi all, I'm trying to test a flow that imports over a module from the contents of another file. When I run the flow locally it works fine but when I try to deploy it to the cloud (on 2.0b12) I receive an error that the module does not exist. Any tips on how to import modules from another file (in the same folder or another folder)?
โœ… 1
b

Bianca Hoch

07/26/2022, 4:12 PM
Hi John, here's some documentation that may be useful for packaging dependencies. See the "Flow packaging and deployment" section.
We also received your email, but we can continue the conversation here if you'd like.
j

John Kang

07/26/2022, 4:18 PM
Bianca, thanks for sharing. I'm in the section "Flow packaging and deployment" and am confused because I'm not trying to package the flow to use somewhere else. Do I still need to package it if I have relative imports to other files?
Also, I've tried packaging using
Copy code
Deployment(
    name="leonardo-deployment",
    flow="./prefect_test.py",
    tags=["tutorial", "test"],
    parameters={"name": "Leo"},
    packager=OrionPackager(serializer=PickleSerializer()),
)
and it throws the error ModuleNotFoundError: No module named 'main_python_files'
When I try to package using
Copy code
Deployment(
    name="leonardo-deployment",
    flow="./prefect_test.py",
    tags=["tutorial", "test"],
    parameters={"name": "Leo"},
packager=DockerPackager(python_environment=CondaEnvironment.from_environment()),
)
it throws a different error Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 843, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "prefect_test_deployment.py", line 32, in <module> packager=DockerPackager(python_environment=CondaEnvironment.from_environment()), File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\prefect\software\conda.py", line 126, in from_environment current_environment_requirements( File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\prefect\software\pip.py", line 70, in current_environment_requirements requirements.append(PipRequirement.from_distribution(dist)) File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\prefect\software\pip.py", line 29, in from_distribution return cls.validate(dist.as_requirement()) File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\pkg_resources\__init__.py", line 2847, in as_requirement return Requirement.parse(spec) File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\pkg_resources\__init__.py", line 3134, in parse req, = parse_requirements(s) File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\pkg_resources\__init__.py", line 3089, in init super(Requirement, self).__init__(requirement_string) File "C:\ProgramData\Anaconda3\envs\Capacity_venv\lib\site-packages\pkg_resources\_vendor\packaging\requirements.py", line 104, in init raise InvalidRequirement( pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'-ywinpty'": Expected W:(abcd...)
b

Bianca Hoch

07/26/2022, 4:38 PM
Hi John, thank you for clarifying. We'll take a look at this to see what could be going on here.
j

John Kang

07/26/2022, 4:39 PM
Is there a way that I can deploy a flow on the cloud even though I'm using relative imports without packaging?
@Bianca Hoch I saw on github that there is some commentary regarding something similar (https://github.com/PrefectHQ/prefect/issues/5984). Do you know if there is a resolution? madkinsz commented 17 days ago When the flow script is run directly, we lose the ability to infer the module name since it's replaced with
__main__
if you call
python <your_script.py>
or
__prefect_loader__
if we run the script for you. If you move your deployment into a separate file and import the flow, we will be able to use that as the flow's import path:
Copy code
from my_flows import kubes_flow

Deployment(
    flow=kubes_flow,
)
I'll try to find a way to determine the module path when they're in the same file. Also note, if your module is a relative import rather than an installed module, this will fail while using the CLI. I'll investigate a fix for that, but you can call
Deployment(...).create()
directly and run the script with Python to get past it.
a

Anna Geller

07/26/2022, 4:48 PM
We are working on that as we speak. Some exciting changes are coming in the GA release tomorrow that will significantly mitigate this problem. Watch the #announcements
Also, follow discourse - we're gonna publish official CI recipes there in the next couple of weeks
j

John Kang

07/26/2022, 4:53 PM
@Anna Geller Thanks for the heads up! I'm looking forward to learning more during the GA release tomorrow. @Bianca Hoch I was able to register the deployment directly from the script by calling the deployment:
Copy code
Deployment(
    name="leonardo-deployment",
    flow="./prefect_test.py",
    tags=["tutorial", "test"],
    parameters={"name": "Leo"},
).create()
Using the create() method. I can run it this way! It works. Thanks for the help
๐Ÿš€ 2
๐ŸŽ‰ 3
Also, I'm going to be at the PACC (Prefect Associate Certification Course) this week so hopefully we get to cover all of these updates!
โญ 2
j

Jeremiah

07/26/2022, 6:09 PM
Thatโ€™s great to hear @John Kang! Itโ€™s going to be an exciting week ๐Ÿ™‚
๐Ÿ™Œ 4
j

John Kang

07/26/2022, 6:40 PM
@Jeremiah Yes sir! Looking forward to it!
k

Khuyen Tran

07/27/2022, 9:56 PM
I will go to PAC as well! Look forward to seeing you tonight @John Kang
๐Ÿ™Œ 1
8 Views