Thread
#prefect-community
    Luis Aguirre

    Luis Aguirre

    7 months ago
    I'm trying to run a simple flow on a jupyter notebook (my goal is to run flows on a jupyter kernel) and getting some errors
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
        246 # Convert the call args/kwargs to a parameter dict
        247 parameters = get_call_parameters(self.fn, args, kwargs)
    --> 249 return enter_flow_run_engine_from_flow_call(self, parameters)
    
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/engine.py:108, in enter_flow_run_engine_from_flow_call(flow, parameters)
        105             return portal.call(begin_run)
        106     else:
        107         # An event loop is not running so we will create one
    --> 108         return anyio.run(begin_run)
        110 # Sync subflow run
        111 if not parent_flow_run_context.flow.isasync:
    
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/anyio/_core/_eventloop.py:56, in run(func, backend, backend_options, *args)
         54 try:
         55     backend_options = backend_options or {}
    ---> 56     return asynclib.run(func, *args, **backend_options)  # type: ignore
         57 finally:
         58     if token:
    
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py:233, in run(func, debug, use_uvloop, policy, *args)
        230         del _task_states[task]
        232 _maybe_set_event_loop_policy(policy, use_uvloop)
    --> 233 return native_run(wrapper(), debug=debug)
    
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/asyncio/runners.py:33, in run(main, debug)
          9 """Execute the coroutine and return the result.
         10 
         11 This function runs the passed coroutine, taking care of
       (...)
         30     asyncio.run(main())
         31 """
         32 if events._get_running_loop() is not None:
    ---> 33     raise RuntimeError(
         34         "asyncio.run() cannot be called from a running event loop")
         36 if not coroutines.iscoroutine(main):
         37     raise ValueError("a coroutine was expected, got {!r}".format(main))
    
    RuntimeError: asyncio.run() cannot be called from a running event loop
    on
    2.0a11
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
        246 # Convert the call args/kwargs to a parameter dict
        247 parameters = get_call_parameters(self.fn, args, kwargs)
    --> 249 return enter_flow_run_engine_from_flow_call(self, parameters)
    
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/engine.py:91, in enter_flow_run_engine_from_flow_call(flow, parameters)
         88 parent_flow_run_context = FlowRunContext.get()
         89 is_subflow_run = parent_flow_run_context is not None
    ---> 91 profile = prefect.context.get_profile_context()
         92 profile.initialize()
         94 begin_run = partial(
         95     create_and_begin_subflow_run if is_subflow_run else create_then_begin_flow_run,
         96     flow=flow,
         97     parameters=parameters,
         98 )
    
    File ~/anaconda3/envs/optimus-py3.8/lib/python3.8/site-packages/prefect/context.py:282, in get_profile_context()
        279 profile_ctx = ProfileContext.get()
        281 if not profile_ctx:
    --> 282     raise MissingContextError("No profile context found.")
        284 return profile_ctx
    
    MissingContextError: No profile context found.
    on
    2.0a12
    Anna Geller

    Anna Geller

    7 months ago
    Ok, I was trying to reproduce. I had existing conda env with Orion 2.0a12 and I added jupyterlab:
    conda install -c conda-forge jupyterlab
    I started it from the same conda virtual env:
    jupyter lab
    Then I tried running the repo trending check sample flow:
    import requests
    from prefect import task, flow, get_run_logger
    
    
    @task
    def extract_url_content(url, params=None):
        return requests.get(url, params=params).content
    
    
    @task
    def check_if_trending(trending_page, repo="prefect"):
        logger = get_run_logger()
        is_trending = repo.encode() in trending_page
        is_phrase = "Nope" if not is_trending else "Yes!"
        <http://logger.info|logger.info>("Is %s tranding? %s", repo, is_phrase)
        return is_trending
    
    
    @flow(name="repo_trending_check")
    def repo_trending_check(
        repo="prefect", url="<https://github.com/trending/python>", window="daily"
    ):
        content = extract_url_content(url, params={"since": window})
        return check_if_trending(content, repo)
    
    
    repo_trending_check()
    And it seems to be working fine. Did you run some async flows? Because async is a bit more complicated due to event loop, so for async flows, it may be easier to start it from the terminal. But anyways, curious to hear more about your use case and see if you can replicate the same setup that worked for me - see the image:
    to be thorough, I should have replicated on 3.8, but my venv was 3.9 (it shouldn't matter but want to note that)
    Luis Aguirre

    Luis Aguirre

    7 months ago
    This is the version of notebook I'm using: About Jupyter Notebook Server Information: You are using Jupyter notebook. The version of the notebook server is: 6.4.6 The server is running on this version of Python:
    Python 3.8.10 (default, May 19 2021, 18:05:58) 
    [GCC 7.3.0]
    Current Kernel Information:
    Python 3.8.10 (default, May 19 2021, 18:05:58) 
    Type 'copyright', 'credits' or 'license' for more information
    IPython 8.0.1 -- An enhanced Interactive Python. Type '?' for help.
    Tried it on Jupyter Lab and had the same issue: Jupyter Lab: Version 3.2.9
    jupyter-client                7.1.0
    jupyter-core                  4.9.1
    jupyter-kernel-gateway        2.5.1
    jupyter-server                1.13.1
    jupyter-server-proxy          3.2.0
    jupyterlab                    3.2.9
    jupyterlab-pygments           0.1.2
    jupyterlab-server             2.10.3
    jupyterlab-widgets            1.0.0
    Anna Geller

    Anna Geller

    7 months ago
    how did you install Orion and Jupyter lab? did you use pip for both? really weird because I also used conda and it worked - I'm on MacOS - are you on ubuntu? can you try deleting your Orion DB file and restart Orion and try again? Orion db is located in your ~/.prefect/orion.db
    Also, can you restart your Jupyter kernel before starting this again? sometimes restarting jupyter fixes such issues
    Luis Aguirre

    Luis Aguirre

    7 months ago
    I'm on a ubuntu server, just uninstalled and installed jupyter and prefect
    Python 3.8.10
    
    jupyter-client                7.1.2
    jupyter-core                  4.9.2
    jupyter-server                1.13.5
    jupyter-server-proxy          3.2.1
    jupyterlab                    3.2.9
    jupyterlab-pygments           0.1.2
    jupyterlab-server             2.10.3
    vaex-jupyter                  0.6.0
    
    prefect                       2.0a11
    
    async-generator               1.10
    async-timeout                 3.0.1
    asyncpg                       0.25.0
    
    ipython                       8.0.1
    Even a freshly started jupyter kernel raises the same error
    I'm on Ubuntu 18.04.5 LTS (Bionic Beaver)
    Anna Geller

    Anna Geller

    7 months ago
    can you share the exact commands you used to install those packages and create a virtual environment? I want to know e.g. whether you used pip or conda etc. Every detail may potentially help reproduce the issue. Also, did the error occur with the exact same code I shared or did you change it to async flow and async tasks?
    Luis Aguirre

    Luis Aguirre

    7 months ago
    Used conda to install jupyter and pip to install prefect, will create a new environment to record the exact commands. The error occurs with the exact same code you shared.
    conda create --name op python=3.8
    conda activate op
    conda install -c conda-forge jupyterlab
    pip install -U "prefect==2.0a11"
    jupyter lab --allow-root --ip=0.0.0.0
    Anna Geller

    Anna Geller

    7 months ago
    Thanks for a thorough check! Can you try this?
    conda create --name op python=3.9
    conda activate op
    pip install -U "prefect>=2.0a12"
    conda install -c conda-forge jupyterlab
    jupyter lab # use default settings instead of this: --allow-root --ip=0.0.0.0
    
    # in a separate terminal window:
    conda activate op
    prefect orion start
    Luis Aguirre

    Luis Aguirre

    7 months ago
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    Input In [1], in <module>
         26     content = extract_url_content(url, params={"since": window})
         27     return check_if_trending(content, repo)
    ---> 30 repo_trending_check()
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
        246 # Convert the call args/kwargs to a parameter dict
        247 parameters = get_call_parameters(self.fn, args, kwargs)
    --> 249 return enter_flow_run_engine_from_flow_call(self, parameters)
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/engine.py:113, in enter_flow_run_engine_from_flow_call(flow, parameters)
        110             return portal.call(begin_run)
        111     else:
        112         # An event loop is not running so we will create one
    --> 113         return anyio.run(begin_run)
        115 # Sync subflow run
        116 if not parent_flow_run_context.flow.isasync:
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/anyio/_core/_eventloop.py:56, in run(func, backend, backend_options, *args)
         54 try:
         55     backend_options = backend_options or {}
    ---> 56     return asynclib.run(func, *args, **backend_options)
         57 finally:
         58     if token:
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/anyio/_backends/_asyncio.py:233, in run(func, debug, use_uvloop, policy, *args)
        230         del _task_states[task]
        232 _maybe_set_event_loop_policy(policy, use_uvloop)
    --> 233 return native_run(wrapper(), debug=debug)
    
    File /opt/miniconda3/envs/op/lib/python3.9/asyncio/runners.py:33, in run(main, debug)
          9 """Execute the coroutine and return the result.
         10 
         11 This function runs the passed coroutine, taking care of
       (...)
         30     asyncio.run(main())
         31 """
         32 if events._get_running_loop() is not None:
    ---> 33     raise RuntimeError(
         34         "asyncio.run() cannot be called from a running event loop")
         36 if not coroutines.iscoroutine(main):
         37     raise ValueError("a coroutine was expected, got {!r}".format(main))
    
    RuntimeError: asyncio.run() cannot be called from a running event loop
    Couldn't omit the --ip=0.0.0.0 argument since I'm running this on a server and I wouldn't be able to open the jupyter lab app
    I think I missed something before, what I'm getting is:
    ---------------------------------------------------------------------------
    MissingContextError                       Traceback (most recent call last)
    Input In [2], in <module>
    ----> 1 repo_trending_check()
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
        246 # Convert the call args/kwargs to a parameter dict
        247 parameters = get_call_parameters(self.fn, args, kwargs)
    --> 249 return enter_flow_run_engine_from_flow_call(self, parameters)
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/engine.py:91, in enter_flow_run_engine_from_flow_call(flow, parameters)
         88 parent_flow_run_context = FlowRunContext.get()
         89 is_subflow_run = parent_flow_run_context is not None
    ---> 91 profile = prefect.context.get_profile_context()
         92 profile.initialize()
         94 begin_run = partial(
         95     create_and_begin_subflow_run if is_subflow_run else create_then_begin_flow_run,
         96     flow=flow,
         97     parameters=parameters,
         98 )
    
    File /opt/miniconda3/envs/op/lib/python3.9/site-packages/prefect/context.py:282, in get_profile_context()
        279 profile_ctx = ProfileContext.get()
        281 if not profile_ctx:
    --> 282     raise MissingContextError("No profile context found.")
        284 return profile_ctx
    
    MissingContextError: No profile context found.
    With version
    2.0a11
    I manage to run it using
    await <http://asyncio.to|asyncio.to>_thread(repo_trending_check)
    Anna Geller

    Anna Geller

    7 months ago
    I see, I believe the fact that you are running your Jupyter on a separate server is the culprit because Orion may not know how to talk to it. Could you try running your Jupyter lab on the same machine as your Orion and see if that works? If so, we would need to figure out only the networking part rather than discussing package dependencies 🙂
    Luis Aguirre

    Luis Aguirre

    7 months ago
    Both Jupyter and Orion are running on the same machine, what I'm noticing is that the issue happens when there's a running event loop ipython:
    nbterm (which is a jupyter notebook clone that runs locally on the terminal
    Results on ipython and on nbterm
    Anna Geller

    Anna Geller

    7 months ago
    Ok, thanks. Let me ask the team since I don't know why you can run it in iPython but not in your Jupyter notebook.
    @Luis Aguirre my colleague encountered the no profile context too with an async task but it only occurs when the cell is run twice. Sharing just FYI:
    ---------------------------------------------------------------------------
    MissingContextError                       Traceback (most recent call last)
    Input In [3], in <module>
         12     data = await blob_storage_download(
         13         blob="prefect.txt",
         14         container="prefect",
         15         azure_credentials=azure_credentials,
         16     )
         17     return data
    ---> 19 await example_blob_storage_download_flow()
    
    File ~/Applications/python/orion/src/prefect/flows.py:249, in Flow.__call__(self, *args, **kwargs)
        246 # Convert the call args/kwargs to a parameter dict
        247 parameters = get_call_parameters(self.fn, args, kwargs)
    --> 249 return enter_flow_run_engine_from_flow_call(self, parameters)
    
    File ~/Applications/python/orion/src/prefect/engine.py:91, in enter_flow_run_engine_from_flow_call(flow, parameters)
         88 parent_flow_run_context = FlowRunContext.get()
         89 is_subflow_run = parent_flow_run_context is not None
    ---> 91 profile = prefect.context.get_profile_context()
         92 profile.initialize()
         94 begin_run = partial(
         95     create_and_begin_subflow_run if is_subflow_run else create_then_begin_flow_run,
         96     flow=flow,
         97     parameters=parameters,
         98 )
    
    File ~/Applications/python/orion/src/prefect/context.py:282, in get_profile_context()
        279 profile_ctx = ProfileContext.get()
        281 if not profile_ctx:
    --> 282     raise MissingContextError("No profile context found.")
        284 return profile_ctx
    
    MissingContextError: No profile context found.
    Luis Aguirre

    Luis Aguirre

    6 months ago
    Hi, is there any update of this?
    Anna Geller

    Anna Geller

    6 months ago
    Did you try to upgrade to the latest version? We introduced some changes to the profiles so this may be resolved if you reinstall now actually
    Luis Aguirre

    Luis Aguirre

    6 months ago
    Yes, thank you, It still raises
    MissingContextError: No profile context found.
    Anna Geller

    Anna Geller

    6 months ago
    Can you perhaps run your flows from the terminal for now? I think running Orion in a Jupyter notebook is not commonly used and not tested yet.
    Luis Aguirre

    Luis Aguirre

    6 months ago
    My goal is to use Orion on a web application that uses a Jupyter Kernel Gateway to process dataframe data. I'm open to recommendations to stop using jupyter kernels for this web-app
    I tried using the API from prefect and managed to run some code but realized the responses with pickled data aren't JavaScript friendly, since I'm using a Node.js back-end
    Anna Geller

    Anna Geller

    6 months ago
    I honestly don't know any more how to help here since I couldn't reproduce the issue on my end. I open an issue here https://github.com/PrefectHQ/prefect/issues/5533 Feel free to add any comments directly to this issue if you have any updates on this