https://prefect.io logo
Title
n

Nathan Ballou

08/01/2022, 7:52 PM
Hi! I'm running into an error when trying to run a Prefect 2.0 flow using Comet.ml and a Ray task runner. The flow seems to fail at the comet import statement:
File "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
    from comet_ml import config
  File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
    from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'
Any thoughts?
The flow runs fine when running locally, but fails when trying to run as a deployment.
n

Nate

08/01/2022, 8:05 PM
I'd imagine you're missing that python dependency in your deployment's runtime whats your runtime infrastructure? and how are you specifying your deployment?
n

Nathan Ballou

08/01/2022, 8:11 PM
I'm running on the same infrastructure as when I ran it locally. I am just running an agent on my local machine and then trying to run the deployment. Are there other steps required to specify?
Everything works fine when using the ConcurrentTaskRunner. It's only when using the RayTaskRunner than I get errors.
n

Nate

08/01/2022, 8:14 PM
hmm okay, this may be an issue with
RayTaskRunner
then - will try to reproduce
n

Nathan Ballou

08/02/2022, 3:56 PM
Hi @Nate! Any progress on this? This is breaking for our use case and we really need to get it resolved. Thanks!
n

Nate

08/02/2022, 4:38 PM
Hi @Nathan Ballou, I spent some time this morning trying to replicate and wasn't able to get your exact error, but did encounter a seemingly similar dependency error did you see
(begin_task_run pid=72046) ModuleNotFoundError: No module named 'packaging.requirements'
at any point in the stack trace while running your deployment with
RayTaskRunner
? either way, I'll probably raise this as a GitHub issue and will link it here if you want to follow along with its progress
n

Nathan Ballou

08/02/2022, 4:41 PM
I didn't see anything like that, but it does seem similar. I'll see if I can make a minimal example. Thanks!
This results in the error (but only if it is a deployment. Running locally runs fine):
from prefect import task, flow
from prefect_ray.task_runners import RayTaskRunner


@task
def test_1():
    print("hello")


@task
def test_2(i):
    import comet_ml

    return i


@flow(task_runner=RayTaskRunner)
def run_tasks():
    test_1()
    output = [test_2(i) for i in range(10)]
    print(output)


if __name__ == "__main__":
    run_tasks()
As a note, it appears to only error when importing comet within the function. Importing outside the RayTaskRunner does not seem to be a problem.
n

Nate

08/02/2022, 5:00 PM
@Nathan Ballou that's great context, thank you! I'll check with the team and figure out why there'd be issues with importing some package like
comet_ml
within a task definition
👍 1
n

Nathan Ballou

08/02/2022, 5:06 PM
Also, I tried moving the import outside of the task in my full code, and got the
packaging.requiremments
error. Seems similar to https://github.com/PrefectHQ/prefect/issues/5976#issuecomment-1201995890
n

Nate

08/02/2022, 5:17 PM
@Nathan Ballou have you tried passing your dependencies to the taskrunner using the
runtime_env
field of
init_kwargs
? that could look like
@flow(
    task_runner=RayTaskRunner(
        init_kwargs={"runtime_env": {"pip": ["prefect==2.0.1, comet_ml, everett, ..."]}},
    )
)
def your_flow():
...
n

Nathan Ballou

08/02/2022, 5:17 PM
Not yet. I'll give that a try
If I try that in the small example, I still get the same error. Again, only within a deployment.
n

Nate

08/02/2022, 6:50 PM
hmm okay, I'll pass this along
n

Nathan Ballou

08/03/2022, 6:56 PM
Hi @Nate! Any update on this?
n

Nate

08/03/2022, 8:25 PM
hi @Nathan Ballou, not yet there's a lot of things we're trying to keep up with following the new release and limited internal bandwidth - will surface progress on this issue here
n

Nathan Ballou

08/03/2022, 8:29 PM
No problem. Thanks for the updates!
👍 1
n

Nate

08/03/2022, 8:55 PM
hey @Nathan Ballou you're on windows yes?
n

Nathan Ballou

08/03/2022, 9:04 PM
Yes
n

Nate

08/03/2022, 9:29 PM
ok thank you