Channels
#prefect-community
Title
n
Nathan Ballou
08/01/2022, 7:52 PMHi! I'm running into an error when trying to run a Prefect 2.0 flow using Comet.ml and a Ray task runner. The flow seems to fail at the comet import statement:
Copy code
File "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
from comet_ml import config
File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'The flow runs fine when running locally, but fails when trying to run as a deployment.
n
Nate
08/01/2022, 8:05 PMI'd imagine you're missing that python dependency in your deployment's runtime
whats your runtime infrastructure? and how are you specifying your deployment?
n
Nathan Ballou
08/01/2022, 8:11 PMI'm running on the same infrastructure as when I ran it locally. I am just running an agent on my local machine and then trying to run the deployment. Are there other steps required to specify?
Everything works fine when using the ConcurrentTaskRunner. It's only when using the RayTaskRunner than I get errors.
n
Nate
08/01/2022, 8:14 PMhmm okay, this may be an issue with
RayTaskRunnern
Nathan Ballou
08/02/2022, 3:56 PMHi @Nate! Any progress on this? This is breaking for our use case and we really need to get it resolved. Thanks!
n
Nate
08/02/2022, 4:38 PMHi @Nathan Ballou, I spent some time this morning trying to replicate and wasn't able to get your exact error, but did encounter a seemingly similar dependency error
did you see
Copy code
(begin_task_run pid=72046) ModuleNotFoundError: No module named 'packaging.requirements'RayTaskRunnern
Nathan Ballou
08/02/2022, 4:41 PMI didn't see anything like that, but it does seem similar. I'll see if I can make a minimal example. Thanks!
This results in the error (but only if it is a deployment. Running locally runs fine):
Copy code
from prefect import task, flow
from prefect_ray.task_runners import RayTaskRunner
@task
def test_1():
print("hello")
@task
def test_2(i):
import comet_ml
return i
@flow(task_runner=RayTaskRunner)
def run_tasks():
test_1()
output = [test_2(i) for i in range(10)]
print(output)
if __name__ == "__main__":
run_tasks()As a note, it appears to only error when importing comet within the function. Importing outside the RayTaskRunner does not seem to be a problem.
n
Nate
08/02/2022, 5:00 PM@Nathan Ballou that's great context, thank you!
I'll check with the team and figure out why there'd be issues with importing some package like
comet_ml👍 1
n
Nathan Ballou
08/02/2022, 5:06 PMAlso, I tried moving the import outside of the task in my full code, and got the
packaging.requiremmentsn
Nate
08/02/2022, 5:17 PM@Nathan Ballou have you tried passing your dependencies to the taskrunner using the
runtime_envinit_kwargs Copy code
@flow(
task_runner=RayTaskRunner(
init_kwargs={"runtime_env": {"pip": ["prefect==2.0.1, comet_ml, everett, ..."]}},
)
)
def your_flow():
...n
Nathan Ballou
08/02/2022, 5:17 PMNot yet. I'll give that a try
If I try that in the small example, I still get the same error. Again, only within a deployment.
n
Nate
08/02/2022, 6:50 PMhmm okay, I'll pass this along
n
Nathan Ballou
08/03/2022, 6:56 PMHi @Nate! Any update on this?
n
Nate
08/03/2022, 8:25 PMhi @Nathan Ballou, not yet
there's a lot of things we're trying to keep up with following the new release and limited internal bandwidth - will surface progress on this issue here
n
Nathan Ballou
08/03/2022, 8:29 PMNo problem. Thanks for the updates!
👍 1
n
Nate
08/03/2022, 8:55 PMhey @Nathan Ballou you're on windows yes?
n
Nathan Ballou
08/03/2022, 9:04 PMYes
n
Nate
08/03/2022, 9:29 PMok thank you
4 Views