https://prefect.io
Join Slack
Hi! I'm running into an error when trying to run a Prefect 2.0 flow using <Comet.ml> and a Ray task ...
# ask-community
n
Hi! I'm running into an error when trying to run a Prefect 2.0 flow using Comet.ml and a Ray task runner. The flow seems to fail at the comet import statement:
Copy code
File "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
    from comet_ml import config
  File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
    from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'
Any thoughts?
The flow runs fine when running locally, but fails when trying to run as a deployment.
n
I'd imagine you're missing that python dependency in your deployment's runtime whats your runtime infrastructure? and how are you specifying your deployment?
n
I'm running on the same infrastructure as when I ran it locally. I am just running an agent on my local machine and then trying to run the deployment. Are there other steps required to specify?
Everything works fine when using the ConcurrentTaskRunner. It's only when using the RayTaskRunner than I get errors.
n
hmm okay, this may be an issue with 
RayTaskRunner
then - will try to reproduce
n
Hi @Nate! Any progress on this? This is breaking for our use case and we really need to get it resolved. Thanks!
n
Hi @Nathan Ballou, I spent some time this morning trying to replicate and wasn't able to get your exact error, but did encounter a seemingly similar dependency error did you see
Copy code
(begin_task_run pid=72046) ModuleNotFoundError: No module named 'packaging.requirements'
at any point in the stack trace while running your deployment with 
RayTaskRunner
? either way, I'll probably raise this as a GitHub issue and will link it here if you want to follow along with its progress
n
I didn't see anything like that, but it does seem similar. I'll see if I can make a minimal example. Thanks!
This results in the error (but only if it is a deployment. Running locally runs fine):
Copy code
from prefect import task, flow
from prefect_ray.task_runners import RayTaskRunner


@task
def test_1():
    print("hello")


@task
def test_2(i):
    import comet_ml

    return i


@flow(task_runner=RayTaskRunner)
def run_tasks():
    test_1()
    output = [test_2(i) for i in range(10)]
    print(output)


if __name__ == "__main__":
    run_tasks()
As a note, it appears to only error when importing comet within the function. Importing outside the RayTaskRunner does not seem to be a problem.
n
@Nathan Ballou that's great context, thank you! I'll check with the team and figure out why there'd be issues with importing some package like 
comet_ml
within a task definition
👍 1
n
Also, I tried moving the import outside of the task in my full code, and got the 
packaging.requiremments
error. Seems similar to https://github.com/PrefectHQ/prefect/issues/5976#issuecomment-1201995890
n
@Nathan Ballou have you tried passing your dependencies to the taskrunner using the 
runtime_env
field of 
init_kwargs
? that could look like
Copy code
@flow(
    task_runner=RayTaskRunner(
        init_kwargs={"runtime_env": {"pip": ["prefect==2.0.1, comet_ml, everett, ..."]}},
    )
)
def your_flow():
...
n
Not yet. I'll give that a try
If I try that in the small example, I still get the same error. Again, only within a deployment.
n
hmm okay, I'll pass this along
n
Hi @Nate! Any update on this?
n
hi @Nathan Ballou, not yet there's a lot of things we're trying to keep up with following the new release and limited internal bandwidth - will surface progress on this issue here
n
No problem. Thanks for the updates!
👍 1
n
hey @Nathan Ballou you're on windows yes?
n
Yes
n
ok thank you
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by