https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Shane Hu

10/18/2022, 2:13 AM
"Can prefect execute a flow in a single thread? See comments for more details and context, I’m trying to get Prefect to run Playwright, but it is single-threaded only." Hi Can Playwright be combined with Prefect?
Copy code
[tool.poetry.dependencies]
python = "^3.10"
prefect = "^2.6.1"
playwright = "^1.27.1"
Copy code
import prefect
from playwright.sync_api import Page, sync_playwright

@prefect.task
def go_to_a_web(page: Page):
    page.goto("<https://formulae.brew.sh/cask/slack>")


@prefect.flow
def main_flow():
    with sync_playwright() as pw:
        browser = pw.chromium.launch(headless=False)
        page = browser.new_page()
        go_to_a_web(page)


main_flow()
I got
Copy code
greenlet.error: cannot switch to a different thread
any help would be much appreciated
1
j

Jarvis Stubblefield

10/18/2022, 8:06 AM
Can you post the full error traceback? I don’t see anything inherently wrong with your code. Have you tried running the code without the
@prefect.flow
and
@prefect.task
decorators? Do you receive the same error?
s

Shane Hu

10/18/2022, 8:34 AM
Thank You for replying. No error without decorators. Full traceback is too long to post
Copy code
16:27:52.985 | ERROR   | Flow run 'macho-porcupine' - Finished in state Failed('Flow run encountered an exception. greenlet.error: cannot switch to a different thread\n')
Traceback (most recent call last):
  File "/Users/me/code/python/test-greenlet/main.py", line 16, in <module>
    main_flow()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/flows.py", line 429, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 150, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/me/.pyenv/versions/3.10.1/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/me/.pyenv/versions/3.10.1/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 229, in create_then_begin_flow_run
    return await state.result(fetch=True)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/states.py", line 74, in _get_state_result
    raise await get_state_exception(state)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 578, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/me/code/python/test-greenlet/main.py", line 14, in main_flow
    go_to_a_web(page)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/tasks.py", line 330, in __call__
    return enter_task_run_engine(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 701, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/Users/me/.pyenv/versions/3.10.1/lib/python3.10/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/Users/me/.pyenv/versions/3.10.1/lib/python3.10/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 842, in get_task_call_return_value
    return await future._result()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/states.py", line 74, in _get_state_result
    raise await get_state_exception(state)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 1190, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/me/code/python/test-greenlet/main.py", line 7, in go_to_a_web
    page.goto("<https://formulae.brew.sh/cask/slack>")
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 8114, in goto
    self._sync(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 102, in _sync
    self._dispatcher_fiber.switch()
greenlet.error: cannot switch to a different thread
I found an issue that probably the same scenario. https://github.com/microsoft/playwright-python/issues/1418#issuecomment-1179200859
Copy code
import threading

import prefect
from playwright.sync_api import Page, sync_playwright


@prefect.task
def go_to_a_web(page: Page):
    print(">>>>> IN TASK", threading.get_ident())
    page.goto("<https://formulae.brew.sh/cask/slack>")

@prefect.flow
def main_flow():
    with sync_playwright() as pw:
        print(">>>>> IN flow", threading.get_ident())
        browser = pw.chromium.launch(headless=False)
        page = browser.new_page()
        go_to_a_web(page)

main_flow()
then
Copy code
❯ poetry run python main.py
16:37:33.752 | INFO    | prefect.engine - Created flow run 'umber-caracal' for flow 'main-flow'
>>>>> IN flow 123145470668800
16:37:36.687 | INFO    | Flow run 'umber-caracal' - Created task run 'go_to_a_web-2066d1d2-0' for task 'go_to_a_web'
16:37:36.687 | INFO    | Flow run 'umber-caracal' - Executing 'go_to_a_web-2066d1d2-0' immediately...
>>>>> IN TASK 123145487458304
16:37:36.728 | ERROR   | Task run 'go_to_a_web-2066d1d2-0' - Encountered exception during execution:
Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/test-greenlet-R45Id2Qo-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 1190, in orchestrate_task_run
j

Jarvis Stubblefield

10/18/2022, 9:21 AM
Definitely some good information. I wish I was of more assistance. I’m learning Prefect as I go and I have always found digging into the community helps me understand it better and faster. I’m not sure of a way to get Prefect to not execute in a thread. Maybe ask something more along the lines of “Can prefect execute a flow in a single thread? See comments for more details and context, I’m trying to get Prefect to run Playwright, but it is single-threaded only.” … then in the thread you can link back to this post.
❤️ 2
👍 3
s

Shane Hu

10/18/2022, 9:41 AM
Thank you so much. I just learned how to ask a good question.
z

Zanie

10/18/2022, 3:28 PM
If you write an async task e.g.
async def foo
it’ll run in the main thread.
👍 1
It looks like the issue here is that you are creating the browser in one thread then using it in another. If you did it all within one task or flow this issue should not occur. Similarly, if you wrote it all in async it’d all happen in one thread.
👍 2
We’re investigating improving this experience (@Andrew Brookins)
👍 4