https://prefect.io logo
Join Slack
Powered by
I'm trying to run a simple flow on a jupyter noteb...
# prefect-community
l
I'm trying to run a simple flow on a jupyter notebook (my goal is to run flows on a jupyter kernel) and getting some errors
Copy code
File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
    246 # Convert the call args/kwargs to a parameter dict
    247 parameters = get_call_parameters(self.fn, args, kwargs)
--> 249 return enter_flow_run_engine_from_flow_call(self, parameters)

File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/engine.py:108, in enter_flow_run_engine_from_flow_call(flow, parameters)
    105             return portal.call(begin_run)
    106     else:
    107         # An event loop is not running so we will create one
--> 108         return anyio.run(begin_run)
    110 # Sync subflow run
    111 if not parent_flow_run_context.flow.isasync:

File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/anyio/_core/_eventloop.py:56, in run(func, backend, backend_options, *args)
     54 try:
     55     backend_options = backend_options or {}
---> 56     return asynclib.run(func, *args, **backend_options)  # type: ignore
     57 finally:
     58     if token:

File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py:233, in run(func, debug, use_uvloop, policy, *args)
    230         del _task_states[task]
    232 _maybe_set_event_loop_policy(policy, use_uvloop)
--> 233 return native_run(wrapper(), debug=debug)

File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/asyncio/runners.py:33, in run(main, debug)
      9 """Execute the coroutine and return the result.
     10 
     11 This function runs the passed coroutine, taking care of
   (...)
     30     asyncio.run(main())
     31 """
     32 if events._get_running_loop() is not None:
---> 33     raise RuntimeError(
     34         "asyncio.run() cannot be called from a running event loop")
     36 if not coroutines.iscoroutine(main):
     37     raise ValueError("a coroutine was expected, got {!r}".format(main))

RuntimeError: asyncio.run() cannot be called from a running event loop
on 
2.0a11
Copy code
File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
    246 # Convert the call args/kwargs to a parameter dict
    247 parameters = get_call_parameters(self.fn, args, kwargs)
--> 249 return enter_flow_run_engine_from_flow_call(self, parameters)

File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/engine.py:91, in enter_flow_run_engine_from_flow_call(flow, parameters)
     88 parent_flow_run_context = FlowRunContext.get()
     89 is_subflow_run = parent_flow_run_context is not None
---> 91 profile = prefect.context.get_profile_context()
     92 profile.initialize()
     94 begin_run = partial(
     95     create_and_begin_subflow_run if is_subflow_run else create_then_begin_flow_run,
     96     flow=flow,
     97     parameters=parameters,
     98 )

File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/context.py:282, in get_profile_context()
    279 profile_ctx = ProfileContext.get()
    281 if not profile_ctx:
--> 282     raise MissingContextError("No profile context found.")
    284 return profile_ctx

MissingContextError: No profile context found.
on 
2.0a12
a
Ok, I was trying to reproduce. I had existing conda env with Orion 2.0a12 and I added jupyterlab:
Copy code
conda install -c conda-forge jupyterlab
I started it from the same conda virtual env:
Copy code
jupyter lab
Then I tried running the repo trending check sample flow:
Copy code
import requests
from prefect import task, flow, get_run_logger


@task
def extract_url_content(url, params=None):
    return requests.get(url, params=params).content


@task
def check_if_trending(trending_page, repo="prefect"):
    logger = get_run_logger()
    is_trending = repo.encode() in trending_page
    is_phrase = "Nope" if not is_trending else "Yes!"
    <http://logger.info|logger.info>("Is %s tranding? %s", repo, is_phrase)
    return is_trending


@flow(name="repo_trending_check")
def repo_trending_check(
    repo="prefect", url="<https://github.com/trending/python>", window="daily"
):
    content = extract_url_content(url, params={"since": window})
    return check_if_trending(content, repo)


repo_trending_check()
And it seems to be working fine. Did you run some async flows? Because async is a bit more complicated due to event loop, so for async flows, it may be easier to start it from the terminal. But anyways, curious to hear more about your use case and see if you can replicate the same setup that worked for me - see the image:
to be thorough, I should have replicated on 3.8, but my venv was 3.9 (it shouldn't matter but want to note that)
l
This is the version of notebook I'm using: About Jupyter Notebook Server Information: You are using Jupyter notebook. The version of the notebook server is: 6.4.6 The server is running on this version of Python:
Copy code
Python 3.8.10 (default, May 19 2021, 18:05:58) 
[GCC 7.3.0]
Current Kernel Information:
Copy code
Python 3.8.10 (default, May 19 2021, 18:05:58) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.1 -- An enhanced Interactive Python. Type '?' for help.
Tried it on Jupyter Lab and had the same issue: Jupyter Lab: Version 3.2.9
Copy code
jupyter-client                7.1.0
jupyter-core                  4.9.1
jupyter-kernel-gateway        2.5.1
jupyter-server                1.13.1
jupyter-server-proxy          3.2.0
jupyterlab                    3.2.9
jupyterlab-pygments           0.1.2
jupyterlab-server             2.10.3
jupyterlab-widgets            1.0.0
a
how did you install Orion and Jupyter lab? did you use pip for both? really weird because I also used conda and it worked - I'm on MacOS - are you on ubuntu? can you try deleting your Orion DB file and restart Orion and try again? Orion db is located in your ~/.prefect/orion.db
Also, can you restart your Jupyter kernel before starting this again? sometimes restarting jupyter fixes such issues
l
I'm on a ubuntu server, just uninstalled and installed jupyter and prefect
Copy code
Python 3.8.10

jupyter-client                7.1.2
jupyter-core                  4.9.2
jupyter-server                1.13.5
jupyter-server-proxy          3.2.1
jupyterlab                    3.2.9
jupyterlab-pygments           0.1.2
jupyterlab-server             2.10.3
vaex-jupyter                  0.6.0

prefect                       2.0a11

async-generator               1.10
async-timeout                 3.0.1
asyncpg                       0.25.0

ipython                       8.0.1
Even a freshly started jupyter kernel raises the same error
I'm on Ubuntu 18.04.5 LTS (Bionic Beaver)
a
can you share the exact commands you used to install those packages and create a virtual environment? I want to know e.g. whether you used pip or conda etc. Every detail may potentially help reproduce the issue. Also, did the error occur with the exact same code I shared or did you change it to async flow and async tasks?
l
Used conda to install jupyter and pip to install prefect, will create a new environment to record the exact commands. The error occurs with the exact same code you shared.
Copy code
conda create --name op python=3.8
conda activate op
conda install -c conda-forge jupyterlab
pip install -U "prefect==2.0a11"
jupyter lab --allow-root --ip=0.0.0.0
a
Thanks for a thorough check! Can you try this?
Copy code
conda create --name op python=3.9
conda activate op
pip install -U "prefect>=2.0a12"
conda install -c conda-forge jupyterlab
jupyter lab # use default settings instead of this: --allow-root --ip=0.0.0.0

# in a separate terminal window:
conda activate op
prefect orion start
l
Copy code
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [1], in <module>
     26     content = extract_url_content(url, params={"since": window})
     27     return check_if_trending(content, repo)
---> 30 repo_trending_check()

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
    246 # Convert the call args/kwargs to a parameter dict
    247 parameters = get_call_parameters(self.fn, args, kwargs)
--> 249 return enter_flow_run_engine_from_flow_call(self, parameters)

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/engine.py:113, in enter_flow_run_engine_from_flow_call(flow, parameters)
    110             return portal.call(begin_run)
    111     else:
    112         # An event loop is not running so we will create one
--> 113         return anyio.run(begin_run)
    115 # Sync subflow run
    116 if not parent_flow_run_context.flow.isasync:

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/anyio/_core/_eventloop.py:56, in run(func, backend, backend_options, *args)
     54 try:
     55     backend_options = backend_options or {}
---> 56     return asynclib.run(func, *args, **backend_options)
     57 finally:
     58     if token:

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/anyio/_backends/_asyncio.py:233, in run(func, debug, use_uvloop, policy, *args)
    230         del _task_states[task]
    232 _maybe_set_event_loop_policy(policy, use_uvloop)
--> 233 return native_run(wrapper(), debug=debug)

File /opt/miniconda3/envs/op/lib/python3.9/asyncio/runners.py:33, in run(main, debug)
      9 """Execute the coroutine and return the result.
     10 
     11 This function runs the passed coroutine, taking care of
   (...)
     30     asyncio.run(main())
     31 """
     32 if events._get_running_loop() is not None:
---> 33     raise RuntimeError(
     34         "asyncio.run() cannot be called from a running event loop")
     36 if not coroutines.iscoroutine(main):
     37     raise ValueError("a coroutine was expected, got {!r}".format(main))

RuntimeError: asyncio.run() cannot be called from a running event loop
Couldn't omit the --ip=0.0.0.0 argument since I'm running this on a server and I wouldn't be able to open the jupyter lab app
I think I missed something before, what I'm getting is:
Copy code
---------------------------------------------------------------------------
MissingContextError                       Traceback (most recent call last)
Input In [2], in <module>
----> 1 repo_trending_check()

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
    246 # Convert the call args/kwargs to a parameter dict
    247 parameters = get_call_parameters(self.fn, args, kwargs)
--> 249 return enter_flow_run_engine_from_flow_call(self, parameters)

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/engine.py:91, in enter_flow_run_engine_from_flow_call(flow, parameters)
     88 parent_flow_run_context = FlowRunContext.get()
     89 is_subflow_run = parent_flow_run_context is not None
---> 91 profile = prefect.context.get_profile_context()
     92 profile.initialize()
     94 begin_run = partial(
     95     create_and_begin_subflow_run if is_subflow_run else create_then_begin_flow_run,
     96     flow=flow,
     97     parameters=parameters,
     98 )

File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/context.py:282, in get_profile_context()
    279 profile_ctx = ProfileContext.get()
    281 if not profile_ctx:
--> 282     raise MissingContextError("No profile context found.")
    284 return profile_ctx

MissingContextError: No profile context found.
With version 
2.0a11
I manage to run it using 
await <http://asyncio.to|asyncio.to>_thread(repo_trending_check)
a
I see, I believe the fact that you are running your Jupyter on a separate server is the culprit because Orion may not know how to talk to it. Could you try running your Jupyter lab on the same machine as your Orion and see if that works? If so, we would need to figure out only the networking part rather than discussing package dependencies 🙂
l
Both Jupyter and Orion are running on the same machine, what I'm noticing is that the issue happens when there's a running event loop ipython:
nbterm (which is a jupyter notebook clone that runs locally on the terminal
Results on ipython and on nbterm
a
Ok, thanks. Let me ask the team since I don't know why you can run it in iPython but not in your Jupyter notebook.
👍 1
@Luis Aguirre my colleague encountered the no profile context too with an async task but it only occurs when the cell is run twice. Sharing just FYI:
Copy code
---------------------------------------------------------------------------
MissingContextError                       Traceback (most recent call last)
Input In [3], in <module>
     12     data = await blob_storage_download(
     13         blob="prefect.txt",
     14         container="prefect",
     15         azure_credentials=azure_credentials,
     16     )
     17     return data
---> 19 await example_blob_storage_download_flow()

File ~/Applications/python/orion/src/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
    246 # Convert the call args/kwargs to a parameter dict
    247 parameters = get_call_parameters(self.fn, args, kwargs)
--> 249 return enter_flow_run_engine_from_flow_call(self, parameters)

File ~/Applications/python/orion/src/prefect/engine.py:91, in enter_flow_run_engine_from_flow_call(flow, parameters)
     88 parent_flow_run_context = FlowRunContext.get()
     89 is_subflow_run = parent_flow_run_context is not None
---> 91 profile = prefect.context.get_profile_context()
     92 profile.initialize()
     94 begin_run = partial(
     95     create_and_begin_subflow_run if is_subflow_run else create_then_begin_flow_run,
     96     flow=flow,
     97     parameters=parameters,
     98 )

File ~/Applications/python/orion/src/prefect/context.py:282, in get_profile_context()
    279 profile_ctx = ProfileContext.get()
    281 if not profile_ctx:
--> 282     raise MissingContextError("No profile context found.")
    284 return profile_ctx

MissingContextError: No profile context found.
👍 1
l
Hi, is there any update of this?
a
Did you try to upgrade to the latest version? We introduced some changes to the profiles so this may be resolved if you reinstall now actually
l
Yes, thank you, It still raises 
MissingContextError: No profile context found.
a
Can you perhaps run your flows from the terminal for now? I think running Orion in a Jupyter notebook is not commonly used and not tested yet.
l
My goal is to use Orion on a web application that uses a Jupyter Kernel Gateway to process dataframe data. I'm open to recommendations to stop using jupyter kernels for this web-app
I tried using the API from prefect and managed to run some code but realized the responses with pickled data aren't JavaScript friendly, since I'm using a Node.js back-end
a
I honestly don't know any more how to help here since I couldn't reproduce the issue on my end. I open an issue here https://github.com/PrefectHQ/prefect/issues/5533 Feel free to add any comments directly to this issue if you have any updates on this
16 Views
Open in Slack
PreviousNext
Powered by