Hi! I'm running into an error when trying to run a...
# ask-communityn
Nathan Ballou
08/01/2022, 7:52 PMHi! I'm running into an error when trying to run a Prefect 2.0 flow using Comet.ml and a Ray task runner. The flow seems to fail at the comet import statement:
Copy code
File "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
from comet_ml import config
File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'Nathan Ballou
08/01/2022, 7:54 PMThe flow runs fine when running locally, but fails when trying to run as a deployment.
n
Nate
08/01/2022, 8:05 PMI'd imagine you're missing that python dependency in your deployment's runtime
whats your runtime infrastructure? and how are you specifying your deployment?
n
Nathan Ballou
08/01/2022, 8:11 PMI'm running on the same infrastructure as when I ran it locally. I am just running an agent on my local machine and then trying to run the deployment. Are there other steps required to specify?
Nathan Ballou
08/01/2022, 8:14 PMEverything works fine when using the ConcurrentTaskRunner. It's only when using the RayTaskRunner than I get errors.
n
Nate
08/01/2022, 8:14 PMhmm okay, this may be an issue with
RayTaskRunnern
Nathan Ballou
08/02/2022, 3:56 PMHi @Nate! Any progress on this? This is breaking for our use case and we really need to get it resolved. Thanks!
n
Nate
08/02/2022, 4:38 PMHi @Nathan Ballou, I spent some time this morning trying to replicate and wasn't able to get your exact error, but did encounter a seemingly similar dependency error
did you see
Copy code
(begin_task_run pid=72046) ModuleNotFoundError: No module named 'packaging.requirements'RayTaskRunnern
Nathan Ballou
08/02/2022, 4:41 PMI didn't see anything like that, but it does seem similar. I'll see if I can make a minimal example. Thanks!
Nathan Ballou
08/02/2022, 4:54 PMThis results in the error (but only if it is a deployment. Running locally runs fine):
Copy code
from prefect import task, flow
from prefect_ray.task_runners import RayTaskRunner
@task
def test_1():
print("hello")
@task
def test_2(i):
import comet_ml
return i
@flow(task_runner=RayTaskRunner)
def run_tasks():
test_1()
output = [test_2(i) for i in range(10)]
print(output)
if __name__ == "__main__":
run_tasks()Nathan Ballou
08/02/2022, 4:57 PMAs a note, it appears to only error when importing comet within the function. Importing outside the RayTaskRunner does not seem to be a problem.
n
Nate
08/02/2022, 5:00 PM@Nathan Ballou that's great context, thank you!
I'll check with the team and figure out why there'd be issues with importing some package like
comet_ml👍 1
n
Nathan Ballou
08/02/2022, 5:06 PMAlso, I tried moving the import outside of the task in my full code, and got the
packaging.requiremmentsn
Nate
08/02/2022, 5:17 PM@Nathan Ballou have you tried passing your dependencies to the taskrunner using the
runtime_envinit_kwargs Copy code
@flow(
task_runner=RayTaskRunner(
init_kwargs={"runtime_env": {"pip": ["prefect==2.0.1, comet_ml, everett, ..."]}},
)
)
def your_flow():
...n
Nathan Ballou
08/02/2022, 5:17 PMNot yet. I'll give that a try
Nathan Ballou
08/02/2022, 6:48 PMIf I try that in the small example, I still get the same error. Again, only within a deployment.
n
Nate
08/02/2022, 6:50 PMhmm okay, I'll pass this along
n
Nathan Ballou
08/03/2022, 6:56 PMHi @Nate! Any update on this?
n
Nate
08/03/2022, 8:25 PMhi @Nathan Ballou, not yet
there's a lot of things we're trying to keep up with following the new release and limited internal bandwidth - will surface progress on this issue here
n
Nathan Ballou
08/03/2022, 8:29 PMNo problem. Thanks for the updates!
👍 1
n
Nate
08/03/2022, 8:55 PMhey @Nathan Ballou you're on windows yes?
n
Nathan Ballou
08/03/2022, 9:04 PMYes
n
Nate
08/03/2022, 9:29 PMok thank you
4 Views