<@ULVA73B9P> my devcontainer setup seems to have n...
# ask-marvin
o
@Marvin my devcontainer setup seems to have not to have access to the local storage to temporarly save code before serving it. what do i need to do to allow temp write to /tmp/runner_storage ? | WARNING | prefect.FlowRunCancellingObserver - Consumer task exited with exception Traceback (most recent call last): File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/_observers.py", line 176, in aexit await self._consumer_task File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/_observers.py", line 67, in _consume_events async for event in self._events_subscriber: ...<11 lines>... ) File "/home/labor/.local/lib/python3.13/site-packages/prefect/events/clients.py", line 642, in anext message = orjson.loads(await self._websocket.recv()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/websockets/asyncio/connection.py", line 322, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: no close frame received or sent 062438.198 | ERROR | asyncio - Exception in callback FlowRunCancellingObserver._start_polling_task.<locals>.<lambda>() at /home/labor/.local/lib/python3.13/site-packages/prefect/runner/_observers.py:99 handle: <Handle FlowRunCancellingObserver._start_polling_task.<locals>.<lambda>() at /home/labor/.local/lib/python3.13/site-packages/prefect/runner/_observers.py:99> Traceback (most recent call last): File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run self._context.run(self._callback, *self._args) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/_observers.py", line 103, in <lambda> if task.exception() ~~~~~~~~~~~~~~^^ File "/home/labor/.local/lib/python3.13/site-packages/prefect/utilities/services.py", line 156, in critical_service_loop await anyio.sleep(sleep) File "/home/labor/.local/lib/python3.13/site-packages/anyio/_core/_eventloop.py", line 87, in sleep return await get_async_backend().sleep(delay) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 2369, in sleep await sleep(delay) File "/usr/local/lib/python3.13/asyncio/tasks.py", line 718, in sleep return await future ^^^^^^^^^^^^ asyncio.exceptions.CancelledError + Exception Group Traceback (most recent call last): | File "/uvd-workflow-orchestrator-example-prefect/dev/dev_serve_remote_code_fromgitlab.py", line 30, in <module> | ).serve( | ~~~~~^ | name="private-gitlab-deploy" | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/flows.py", line 1146, in serve | asyncio.run(runner.start(webserver=webserver)) | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run | return runner.run(main) | ~~~~~~~~~~^^^^^^ | File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run | return self._loop.run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete | return future.result() | ~~~~~~~~~~~~~^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 506, in start | async with self as runner: | ^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 1589, in aexit | await self._exit_stack.__aexit__(*exc_info) | File "/usr/local/lib/python3.13/contextlib.py", line 768, in aexit | raise exc | File "/usr/local/lib/python3.13/contextlib.py", line 751, in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in aexit | raise BaseExceptionGroup( | "unhandled errors in a TaskGroup", self._exceptions | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 509, in start | async with self._loops_task_group as loops_task_group: | ^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in aexit | raise BaseExceptionGroup( | "unhandled errors in a TaskGroup", self._exceptions | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/labor/.local/lib/python3.13/site-packages/prefect/utilities/services.py", line 64, in critical_service_loop | await workload() | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/storage.py", line 700, in pull_code | await self._block.get_directory(local_path=str(self.destination)) | File "/home/labor/.local/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 341, in ctx_call | result = await async_fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped | return await copy(fn, *args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 111, in call | do = await self.iter(retry_state=retry_state) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 153, in iter | result = await action(retry_state) | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/_utils.py", line 99, in inner | return call(*args, **kwargs) | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/__init__.py", line 420, in exc_check | raise retry_exc.reraise() | ~~~~~~~~~~~~~~~~~^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/__init__.py", line 187, in reraise | raise self.last_attempt.result() | ~~~~~~~~~~~~~~~~~~~~~~~~^^ | File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result | return self.__get_result() | ~~~~~~~~~~~~~~~~~^^ | File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result | raise self._exception | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 114, in call | result = await fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect_gitlab/repositories.py", line 184, in get_directory | shutil.copytree( | ~~~~~~~~~~~~~~~^ | src=content_source, dst=content_destination, dirs_exist_ok=True | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File "/usr/local/lib/python3.13/shutil.py", line 593, in copytree | return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, | ignore=ignore, copy_function=copy_function, | ignore_dangling_symlinks=ignore_dangling_symlinks, | dirs_exist_ok=dirs_exist_ok) | File "/usr/local/lib/python3.13/shutil.py", line 547, in _copytree | raise Error(errors) | shutil.Error: [('/tmp/tmpa1dba24nprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', '/tmp/runner_storage/51f5364b-017c-4ebb-b1d3-d6a0d00df87a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', "[Errno 13] Permission denied: '/tmp/runner_storage/51f5364b-017c-4ebb-b1d3-d6a0d00df87a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack'"), ('/tmp/tmpa1dba24nprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', '/tmp/runner_storage/51f5364b-017c-4ebb-b1d3-d6a0d00df87a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', "[Errno 13] Permission denied: '/tmp/runner_storage/51f5364b-017c-4ebb-b1d3-d6a0d00df87a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev'"), ('/tmp/tmpa1dba24nprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', '/tmp/runner_storage/51f5364b-017c-4ebb-b1d3-d6a0d00df87a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', "[Errno 13] Permission denied: '/tmp/runner_storage/51f5364b-017c-4ebb-b1d3-d6a0d00df87a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx'")]
m
Your message was too long, here's your message at the allowed limit: my devcontainer setup seems to have not to have access to the local storage to temporarly save code before serving it what do i need to do to allow temp write to /tmp/runner_storage ? | WARNING | prefect FlowRunCancellingObserver - Consumer task exited with exception Traceback (most recent call last) File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py line 176 in aexit await self _consumer_task File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py line 67 in _consume_events async for event in self _events_subscriber <11 lines> ) File /home/labor/ local/lib/python3 13/site-packages/prefect/events/clients py line 642 in anext message = orjson loads(await self _websocket recv()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /home/labor/ local/lib/python3 13/site-packages/websockets/asyncio/connection py line 322 in recv raise self protocol close_exc from self recv_exc websockets exceptions ConnectionClosedError no close frame received or sent 06 24 38 198 | ERROR | asyncio - Exception in callback FlowRunCancellingObserver _start_polling_task <locals> <lambda>() at /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py 99 handle <Handle FlowRunCancellingObserver _start_polling_task <locals> <lambda>() at /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py 99> Traceback (most recent call last) File /usr/local/lib/python3 13/asyncio/events py line 89 in _run self _context run(self _callback *self _args) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py line 103 in <lambda> if task exception() ~~~~~~~~~~~~~~^^ File /home/labor/ local/lib/python3 13/site-packages/prefect/utilities/services py line 156 in critical_service_loop await anyio sleep(sleep) File /home/labor/ local/lib/python3 13/site-packages/anyio/_core/_eventloop py line 87 in sleep return await get_async_backend() sleep(delay) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 2369 in sleep await sleep(delay) File /usr/local/lib/python3 13/asyncio/tasks py line 718 in sleep return await future ^^^^^^^^^^^^ asyncio exceptions CancelledError + Exception Group Traceback (most recent call last) | File /uvd-workflow-orchestrator-example-prefect/dev/dev_serve_remote_code_fromgitlab py line 30 in <module> | ) serve( | ~~~~~^ | name= private-gitlab-deploy | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/flows py line 1146 in serve | asyncio run(runner start(webserver=webserver)) | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 13/asyncio/runners py line 195 in run | return runner run(main) | ~~~~~~~~~~^^^^^^ | File /usr/local/lib/python3 13/asyncio/runners py line 118 in run | return self _loop run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File /usr/local/lib/python3 13/asyncio/base_events py line 725 in run_until_complete | return future result() | ~~~~~~~~~~~~~^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 506 in start | async with self as runner | ^^^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 1589 in aexit | await self _exit_stack __aexit__(*exc_info) | File /usr/local/lib/python3 13/contextlib py line 768 in aexit | raise exc | File /usr/local/lib/python3 13/contextlib py line 751 in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 781 in aexit | raise BaseExceptionGroup( | unhandled errors in a TaskGroup self _exceptions | ) from None | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+-
--------------- 1 ---------------- | Exception Group Traceback (most recent call last) | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 509 in start | async with self _loops_task_group as loops_task_group | ^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 781 in aexit | raise BaseExceptionGroup( | unhandled errors in a TaskGroup self _exceptions | ) from None | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call
o
@Marvin my devcontainer setup seems to have not to have access to the local storage to temporarly save code before serving it what do i need to do to allow temp write to /tmp/runner_storage ? | WARNING | prefect FlowRunCancellingObserver - Consumer task exited with exception Traceback (most recent call last) File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py line 176 in aexit await self _consumer_task File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py line 67 in _consume_events async for event in self _events_subscriber <11 lines> ) File /home/labor/ local/lib/python3 13/site-packages/prefect/events/clients py line 642 in anext message = orjson loads(await self _websocket recv()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /home/labor/ local/lib/python3 13/site-packages/websockets/asyncio/connection py line 322 in recv raise self protocol close_exc from self recv_exc websockets exceptions ConnectionClosedError no close frame received or sent 06 24 38 198 | ERROR | asyncio - Exception in callback FlowRunCancellingObserver _start_polling_task <locals> <lambda>() at /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py 99 handle <Handle FlowRunCancellingObserver _start_polling_task <locals> <lambda>() at /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py 99> Traceback (most recent call last) File /usr/local/lib/python3 13/asyncio/events py line 89 in _run self _context run(self _callback *self _args) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/_observers py line 103 in <lambda> if task exception() ~~~~~~~~~~~~~~^^ File /home/labor/ local/lib/python3 13/site-packages/prefect/utilities/services py line 156 in critical_service_loop await anyio sleep(sleep) File /home/labor/ local/lib/python3 13/site-packages/anyio/_core/_eventloop py line 87 in sleep return await get_async_backend() sleep(delay) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 2369 in sleep await sleep(delay) File /usr/local/lib/python3 13/asyncio/tasks py line 718 in sleep return await future ^^^^^^^^^^^^ asyncio exceptions CancelledError + Exception Group Traceback (most recent call last) | File /uvd-workflow-orchestrator-example-prefect/dev/dev_serve_remote_code_fromgitlab py line 30 in <module> | ) serve( | ~~~~~^ | name= private-gitlab-deploy | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/flows py line 1146 in serve | asyncio run(runner start(webserver=webserver)) | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 13/asyncio/runners py line 195 in run | return runner run(main) | ~~~~~~~~~~^^^^^^ | File /usr/local/lib/python3 13/asyncio/runners py line 118 in run | return self _loop run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File /usr/local/lib/python3 13/asyncio/base_events py line 725 in run_until_complete | return future result() | ~~~~~~~~~~~~~^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 506 in start | async with self as runner | ^^^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 1589 in aexit | await self _exit_stack __aexit__(*exc_info) | File /usr/local/lib/python3 13/contextlib py line 768 in aexit | raise exc | File /usr/local/lib/python3 13/contextlib py line 751 in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 781 in aexit | raise BaseExceptionGroup( | unhandled errors in a TaskGroup self _exceptions | ) from None | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+- [08:33 Uhr] --------------- 1 ---------------- | Exception Group Traceback (most recent call last) | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 509 in start | async with self _loops_task_group as loops_task_group | ^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 781 in aexit | raise BaseExceptionGroup( | unhandled errors in a TaskGroup self _exceptions | ) from None | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------
m
thought for 349.6 seconds
Thanks for the traceback — the short version is: when you serve a flow from source in Prefect 3.x, the runner checks out your code into a temporary directory under ${TMPDIR:-/tmp}/runner_storage/<uuid>. If your devcontainer can’t write there, serving will fail and you’ll see secondary errors like the websocket disconnect. What to do Option A — Fix /tmp permissions (recommended) - In the container, verify /tmp is writable and has the sticky bit:
Copy code
ls -ld /tmp
# should be: drwxrwxrwt (1777)
- If not, fix it (as root):
Copy code
chmod 1777 /tmp
- Ensure /tmp is not mounted read-only or with restrictive options:
Copy code
mount | grep ' /tmp '
- In a devcontainer, you can force a writable tmpfs: - devcontainer.json (new syntax)
Copy code
"mounts": ["type=tmpfs,target=/tmp"]
- or devcontainer.json (older runArgs style)
Copy code
"runArgs": ["--tmpfs", "/tmp:exec,mode=1777"]
Option B — Point Prefect’s temp storage somewhere else Prefect uses Python’s tempfile.gettempdir(), which respects the TMPDIR environment variable. Set TMPDIR to a writable path that lives on a mounted volume (e.g., your workspace). - Create a writable directory and set TMPDIR:
Copy code
mkdir -p /workspaces/your-repo/.tmp && chmod 777 /workspaces/your-repo/.tmp
export TMPDIR=/workspaces/your-repo/.tmp
- To make it persistent in a devcontainer, add to devcontainer.json:
Copy code
"containerEnv": {
  "TMPDIR": "/workspaces/your-repo/.tmp"
},
"postCreateCommand": "mkdir -p /workspaces/your-repo/.tmp && chmod 777 /workspaces/your-repo/.tmp"
Option C — Avoid temp storage by using a local path source If you can clone your repo into the workspace and serve from the local path, Prefect won’t need to check out into /tmp:
Copy code
from prefect import flow

flow = flow.from_source(
    source="/workspaces/your-repo",
    entrypoint="path/to/flow.py:my_flow",
)
flow.serve(name="my-deployment")
Quick checks to confirm the root cause - What temp dir is Prefect using?
Copy code
python -c "import tempfile; print(tempfile.gettempdir())"
- Can you write to the default runner dir?
Copy code
mkdir -p /tmp/runner_storage && touch /tmp/runner_storage/test && echo OK
Reference - Prefect stores flow code pulled from source in a temp directory; for background on the runner’s temp usage see this related discussion that shows the runner path pattern under tempfile.gettempdir() → runner_storage: flow.from_source Git pull/runner_storage context - How to store/serve flow code from source: Store flow code for deployments If you can share: - Your devcontainer.json (or the relevant mounts/runArgs) - Output of
ls -ld /tmp
- Prefect version (
prefect version
) I can suggest the exact minimal change.
o
@Marvin this is strange, i can do this inside my devcontainer, without modyfing the devcontainer but still my script is failing... python -c "import tempfile; print(tempfile.gettempdir())" /tmp labor@dc19b557640a:/uvd-workflow-orchestrator-example-prefect$ mkdir -p /tmp/runner_storage && touch /tmp/runner_storage/test && echo OK OK labor@dc19b557640a:/uvd-workflow-orchestrator-example-prefect$ import os import sys #you must ensure to env this before importing prefect otherwise it will do a local temp. prefect server os.environ["PREFECT_API_URL"] = "http://host.docker.internal:4200/api" os.environ["PREFECT_DEFAULT_WORK_POOL_NAME"] = "local-pool" os.environ["PREFECT_LOGGING_LEVEL"] = "DEBUG" os.environ["GIT_TRACE"] = "1" os.environ["GIT_CURL_VERBOSE"] = "1" os.environ["GIT_SSL_NO_VERIFY"] = "True" sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), '..', 'workflows'))) from prefect import flow from prefect_gitlab.repositories import GitLabRepository from prefect_gitlab import GitLabCredentials if name == "__main__": gitlab_repository_block = GitLabRepository.load("prefectprototypegit") gitlab_credentials_block = GitLabCredentials.load("mycredential") gitlab_repository_block.credentials = gitlab_credentials_block print("error") flow.from_source( source=gitlab_repository_block, entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow", ).serve( name="private-gitlab-deploy" )
m
thought for 69.2 seconds
Thanks for the extra detail — that helps. Since you can write to /tmp/runner_storage manually, permissions on /tmp aren’t the blocker. A few likely culprits based on your snippet and the earlier trace: Top suspects 1) Python 3.13 support - Your traceback shows Python 3.13 paths. Prefect 3.x is currently tested/supported on Python 3.10–3.12. 3.13 can exhibit asyncio/websocket oddities like the ones you’re seeing. - Suggestion: try the same script with Python 3.12 (or 3.11) in your devcontainer. If it works there, we’ve found the root cause. 2) API connectivity from the devcontainer - You’re pointing PREFECT_API_URL at http://host.docker.internal:4200/api. On Linux/devcontainers, that hostname often isn’t resolvable unless you add a host mapping. - Verify connectivity:
Copy code
curl -sf <http://host.docker.internal:4200/api/health> || echo "unreachable"
- If unreachable, add this to your devcontainer.json and rebuild:
Copy code
"runArgs": ["--add-host=host.docker.internal:host-gateway"]
or point to the host’s IP directly (e.g., http://172.17.0.1:4200/api, depending on your setup). 3) Git checkout via the GitLab block failing - You’ve set GIT_TRACE/GIT_CURL_VERBOSE — good. The runner pulls code into ${TMPDIR:-/tmp}/runner_storage/<uuid>. If the clone/pull fails, the runner will shut down quickly and you’ll see secondary websocket/observer errors. - Please run once with DEBUG logs and share the first error above the websocket lines; look for messages from prefect.runner.storage or any subprocess errors involving git. Quick experiments to isolate - Force a known-writable temp dir early (before importing Prefect), just to rule out any odd temp resolution:
Copy code
import os
  os.environ["TMPDIR"] = "/workspaces/your-repo/.tmp"
  os.makedirs(os.environ["TMPDIR"], exist_ok=True)
  os.chmod(os.environ["TMPDIR"], 0o777)
- Confirm API health from inside the container:
Copy code
curl -sf "$PREFECT_API_URL/health" || echo "unreachable"
- Try a simple from_source with a public repo to confirm the flow runner can clone at all:
Copy code
from prefect import flow

  flow = flow.from_source(
      source="<https://github.com/PrefectHQ/prefect.git>",
      entrypoint="examples/flows/hello_world.py:hello_world_flow",
  )
  flow.serve(name="smoke-test")
If this works but your GitLab block fails, we’ll focus on credentials/clone settings. A couple more notes for your script - You do not need to modify sys.path when using flow.from_source with a repository block; the runner loads from the cloned repo path. It shouldn’t break anything, but it’s unnecessary. - Make sure you set all env vars before importing Prefect (you did — good). - If you want to be explicit about the temp base, TMPDIR is the right lever; Prefect honors Python’s tempfile.gettempdir() which respects TMPDIR. Useful references - Store flow code for deployments and serving: https://docs-3.prefect.io/v3/how-to-guides/deployments/store-flow-code - Context on runner temp storage pattern (uses tempfile.gettempdir()/runner_storage): https://github.com/PrefectHQ/prefect/issues/11546 Could you share: - Prefect version (prefect version) and Python version (python --version) - Result of curl -sf "$PREFECT_API_URL/health" - The first error lines in DEBUG logs prior to the websocket close (especially anything about git clone/pull) If you can switch the devcontainer to Python 3.12 and add the host-gateway mapping for host.docker.internal, that will likely resolve this quickly.
o
@Marvin sry i do only have python 3.13, also host.docker.internal is working fine. the issue is a permission denied error... syncio.exceptions.CancelledError + Exception Group Traceback (most recent call last): | File "/uvd-workflow-orchestrator-example-prefect/dev/dev_serve_remote_code_fromgitlab.py", line 29, in <module> | ).serve( | ~~~~~^ | name="private-gitlab-deploy" | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/flows.py", line 1146, in serve | asyncio.run(runner.start(webserver=webserver)) | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run | return runner.run(main) | ~~~~~~~~~~^^^^^^ | File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run | return self._loop.run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete | return future.result() | ~~~~~~~~~~~~~^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 506, in start | async with self as runner: | ^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 1589, in aexit | await self._exit_stack.__aexit__(*exc_info) | File "/usr/local/lib/python3.13/contextlib.py", line 768, in aexit | raise exc | File "/usr/local/lib/python3.13/contextlib.py", line 751, in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in aexit | raise BaseExceptionGroup( | "unhandled errors in a TaskGroup", self._exceptions | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 509, in start | async with self._loops_task_group as loops_task_group: | ^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in aexit | raise BaseExceptionGroup( | "unhandled errors in a TaskGroup", self._exceptions | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/labor/.local/lib/python3.13/site-packages/prefect/utilities/services.py", line 64, in critical_service_loop | await workload() | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/storage.py", line 700, in pull_code | await self._block.get_directory(local_path=str(self.destination)) | File "/home/labor/.local/lib/python3.13/site-packages/prefect/utilities/asyncutils.py", line 341, in ctx_call | result = await async_fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped | return await copy(fn, *args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 111, in call | do = await self.iter(retry_state=retry_state) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 153, in iter | result = await action(retry_state) | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/_utils.py", line 99, in inner | return call(*args, **kwargs) | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/__init__.py", line 420, in exc_check | raise retry_exc.reraise() | ~~~~~~~~~~~~~~~~~^^ | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/__init__.py", line 187, in reraise | raise self.last_attempt.result() | ~~~~~~~~~~~~~~~~~~~~~~~~^^ | File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result | return self.__get_result() | ~~~~~~~~~~~~~~~~~^^ | File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result | raise self._exception | File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 114, in call | result = await fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect_gitlab/repositories.py", line 184, in get_directory | shutil.copytree( | ~~~~~~~~~~~~~~~^ | src=content_source, dst=content_destination, dirs_exist_ok=True | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File "/usr/local/lib/python3.13/shutil.py", line 593, in copytree | return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, | ignore=ignore, copy_function=copy_function, | ignore_dangling_symlinks=ignore_dangling_symlinks, | dirs_exist_ok=dirs_exist_ok) | File "/usr/local/lib/python3.13/shutil.py", line 547, in _copytree | raise Error(errors) | shutil.Error: [('/tmp/tmpcyxyk38qprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', "[Errno 13] Permission denied: '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack'"), ('/tmp/tmpcyxyk38qprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', "[Errno 13] Permission denied: '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev'"), ('/tmp/tmpcyxyk38qprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', "[Errno 13] Permission denied: '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx'")]
m
Your message was too long, here's your message at the allowed limit: sry i do only have python 3 13 also host docker internal is working fine the issue is a permission denied error syncio exceptions CancelledError + Exception Group Traceback (most recent call last) | File /uvd-workflow-orchestrator-example-prefect/dev/dev_serve_remote_code_fromgitlab py line 29 in <module> | ) serve( | ~~~~~^ | name= private-gitlab-deploy | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ) | ^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/flows py line 1146 in serve | asyncio run(runner start(webserver=webserver)) | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 13/asyncio/runners py line 195 in run | return runner run(main) | ~~~~~~~~~~^^^^^^ | File /usr/local/lib/python3 13/asyncio/runners py line 118 in run | return self _loop run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File /usr/local/lib/python3 13/asyncio/base_events py line 725 in run_until_complete | return future result() | ~~~~~~~~~~~~~^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 506 in start | async with self as runner | ^^^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 1589 in aexit | await self _exit_stack __aexit__(*exc_info) | File /usr/local/lib/python3 13/contextlib py line 768 in aexit | raise exc | File /usr/local/lib/python3 13/contextlib py line 751 in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 781 in aexit | raise BaseExceptionGroup( | unhandled errors in a TaskGroup self _exceptions | ) from None | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last) | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/runner py line 509 in start | async with self _loops_task_group as loops_task_group | ^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/anyio/_backends/_asyncio py line 781 in aexit | raise BaseExceptionGroup( | unhandled errors in a TaskGroup self _exceptions | ) from None | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last) | File /home/labor/ local/lib/python3 13/site-packages/prefect/utilities/services py line 64 in critical_service_loop | await workload() | File /home/labor/ local/lib/python3 13/site-packages/prefect/runner/storage py line 700 in pull_code | await self _block get_directory(local_path=str(self destination)) | File /home/labor/ local/lib/python3 13/site-packages/prefect/utilities/asyncutils py line 341 in ctx_call | result = await async_fn(*args *kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/tenacity/asyncio/__init__ py line 189 in async_wrapped | return await copy(fn *args *kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/tenacity/asyncio/__init__ py line 111 in call | do = await self iter(retry_state=retry_state) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/tenacity/asyncio/__init__ py line 153 in iter | result = await action(retry_state) | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/tenacity/_utils py line 99 in inner | return call(*args *kwargs) | File /home/labor/ local/lib/python3 13/site-packages/tenacity/__init__ py line 420 in exc_check | raise retry_exc reraise() | ~~~~~~~~~~~~~~~~~^^ | File /home/labor/ local/lib/python3 13/site-packages/tenacity/__init__ py line 187 in reraise | raise self last_attempt result() | ~~~~~~~~~~~~~~~~~~~~~~~~^^ | File /usr/local/lib/python3 13/concurrent/futures/_base py line 449 in result | return self __get_result() | ~
~~~~~~~~~~~~~~~~^^ | File /usr/local/lib/python3 13/concurrent/futures/_base py line 401 in __get_result | raise self _exception | File /home/labor/ local/lib/python3 13/site-packages/tenacity/asyncio/__init__ py line 114 in call | result = await fn(*args *kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File /home/labor/ local/lib/python3 13/site-packages/prefect_gitlab/repositories py line 184 in get_directory | shutil copytree( | ~~~~~~~~~~~~~~~^ | src=content_source dst=content_destination dirs_exist_ok=True | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
o
@Marvin but it is a permission error, could we rerite the flow to check that it is pulled and not served? | shutil.Error: [('/tmp/tmpcyxyk38qprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', "[Errno 13] Permission denied: '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack'"), ('/tmp/tmpcyxyk38qprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', "[Errno 13] Permission denied: '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev'"), ('/tmp/tmpcyxyk38qprefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', "[Errno 13] Permission denied: '/tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx'")]
m
thought for 57.5 seconds
You’re right — that stack shows a straight permission error while Prefect is copying the repo into its runner storage: shutil.Error: Permission denied writing into /tmp/runner_storage/<uuid>/gitlab-repository-prefectprototypegit/.git/objects/pack/... What’s happening - Prefect clones your repo into a staging temp dir (/tmp/tmp...prefect) then copies it into the runner’s working dir under ${TMPDIR:-/tmp}/runner_storage/<uuid>/... - If that destination path (or parts of it) were created earlier by a different user (or with restrictive perms/umask), the subsequent copy will fail even though you can write to /tmp generally. Quick fixes - Clean stale runner storage and ensure ownership
Copy code
rm -rf /tmp/runner_storage/*
  # or fix ownership if needed
  # sudo chown -R "$(id -u):$(id -g)" /tmp/runner_storage
- Or set TMPDIR to a workspace path you control (ensures fresh, consistent perms)
Copy code
export TMPDIR=/workspaces/your-repo/.tmp
  mkdir -p "$TMPDIR" && chmod 777 "$TMPDIR"
Do this before importing Prefect. Option: Avoid copying .git entirely If your GitLabRepository block supports selecting directories, set it to only copy your flow code (skipping .git):
Copy code
gitlab_repository_block.directories = ["workflows"]
This typically prevents copying .git, which is where your permission error occurs. “Rewrite the flow to check that it is pulled and not served” If your goal is to pre-pull the code to a path you control and then serve from that local path, here’s a pattern that avoids Prefect’s temp copy: - Step 1: Clone the repo yourself into a known-writable location (with your own credentials setup) - Step 2: Serve from that local path Example:
Copy code
import os
import subprocess
from pathlib import Path
from prefect import flow

# Set a safe temp base first
os.environ["TMPDIR"] = "/workspaces/uvd-workflow-orchestrator-example-prefect/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

# 1) Pre-pull into a path you control
repo_dir = Path("/workspaces/uvd-workflow-orchestrator-example-prefect/_src")
if repo_dir.exists():
    # start fresh to avoid stale perms
    subprocess.run(["rm", "-rf", str(repo_dir)], check=True)

# Use your preferred git auth here (token/ssh). Example with HTTPS:
subprocess.run([
    "git", "clone", "--depth", "1",
    "<https://gitlab.example.com/group/project.git>",
    str(repo_dir)
], check=True)

# 2) Serve from the local path
my_flow = flow.from_source(
    source=str(repo_dir),
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
my_flow.serve(name="private-gitlab-deploy")
If you’d like to keep using your GitLabRepository block but ensure fresh perms, add this preflight at the very top (before importing Prefect):
Copy code
import os
from pathlib import Path

os.environ["TMPDIR"] = "/workspaces/uvd-workflow-orchestrator-example-prefect/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

# Clean any stale runner storage
rs_base = Path(os.environ["TMPDIR"]) / "runner_storage"
if rs_base.exists():
    # remove stale dirs that may be owned by another user
    import shutil
    shutil.rmtree(rs_base, ignore_errors=True)
Why your quick test succeeded but Prefect still fails - Touching /tmp/runner_storage/test only proves the top-level directory is writable. Prefect is writing deeper into /tmp/runner_storage/<uuid>/.../.git/objects/pack, which may have been created by a different user/umask in a prior run. That nested directory can block overwrites even when the parent is writable. If you’d like, share the ownership of the failing path:
Copy code
ls -ld /tmp/runner_storage \
       /tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334 \
       /tmp/runner_storage/38726217-2925-408a-95f6-e65d81524334/gitlab-repository-prefectprototypegit/.git/objects/pack
and I’ll suggest the minimal change. Also confirm if setting TMPDIR to a workspace path and cleaning runner_storage resolves it for you.
o
@Marvin so you are suggesting that it happens in the serve command? is there another command, after flow.from_source we can submit to check wther the checkout works?
m
thought for 106.5 seconds
Great question. Yes — the checkout happens when the runner materializes code for a run (triggered by serve when it starts a run). You can test the checkout step directly (without serving) by invoking the storage’s pull_code yourself. Here are two ways to do that: A) Use your GitLabRepository block via BlockStorageAdapter and call pull_code - This preflight will clone into Prefect’s runner storage and let you inspect the destination. - You can then reuse the same storage object in flow.from_source to avoid a second clone.
Copy code
import os
import asyncio
from pathlib import Path

# Make sure this is set before importing Prefect to control temp base
os.environ["TMPDIR"] = "/workspaces/your-repo/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

from prefect.runner.storage import BlockStorageAdapter
from prefect_gitlab import GitLabRepository, GitLabCredentials

async def main():
    repo = GitLabRepository.load("prefectprototypegit")
    creds = GitLabCredentials.load("mycredential")
    repo.credentials = creds

    storage = BlockStorageAdapter(block=repo, pull_interval=60)

    # Preflight: perform the checkout
    await storage.pull_code()

    # Inspect where it landed
    print("Pulled to:", storage.destination)
    # Optional: list files to verify
    import subprocess
    subprocess.run(["ls", "-la", str(storage.destination)], check=False)

asyncio.run(main())
If this succeeds, your checkout path/permissions are good. You can then pass storage into flow.from_source to reuse it:
Copy code
from prefect import flow as flow_cls
flow = flow_cls.from_source(
    source=storage,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
flow.serve(name="private-gitlab-deploy")
B) Use a plain Git URL to isolate the problem - This helps check if the issue is in the GitLab block vs. general runner storage.
Copy code
import os, asyncio, subprocess
from pathlib import Path
os.environ["TMPDIR"] = "/workspaces/your-repo/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

from prefect.runner.storage import create_storage_from_source

async def main():
    storage = create_storage_from_source("<https://github.com/PrefectHQ/prefect.git>")
    await storage.pull_code()
    print("Pulled to:", storage.destination)
    subprocess.run(["ls", "-la", str(storage.destination)], check=False)

asyncio.run(main())
Why this helps - Your error shows shutil copying from a staging tmp dir into /tmp/runner_storage/<uuid>/.../.git/objects/pack and failing with EACCES. Calling pull_code directly will reproduce that permission issue in isolation, without the serve loop masking it with subsequent websocket errors. If pull_code fails with Permission denied: - Clean stale runner storage (created by another user/umask) and try again:
Copy code
rm -rf ${TMPDIR:-/tmp}/runner_storage/*
- Or change TMPDIR to a workspace you own (as in the examples above), then rerun the preflight. Links - Storing/serving flow code: https://docs-3.prefect.io/v3/how-to-guides/deployments/store-flow-code If you share the output of the preflight (especially storage.destination and ls -ld of that path), I can suggest the minimal fix.
o
@Marvin the pull storage cmd is sucessfully creating a storage object in my dev container folder within the ropository at storage.destination. when i do that again, it says Permission denied: '/uvd-workflow-orchestrator-example-prefect/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx'") so what i think is the issue here that in /tmp/runner_storage there is already code, and you cant overwrite it, because of permission errno 13 permission denied. labor@dc19b557640a:/uvd-workflow-orchestrator-example-prefect$ ls -ld /tmp/runner_storage drwxr-xr-x. 14 labor labor 4096 Oct 9 06:47 /tmp/runner_storage labor@dc19b557640a:/uvd-workflow-orchestrator-example-prefect$ ls -ls /tmp/runner_storage total 0 0 drwxr-xr-x. 2 labor labor 6 Oct 8 08:50 089a898e-ffe9-4499-a0c3-3fa4304fb432 0 drwxr-xr-x. 2 labor labor 6 Oct 8 08:59 1dee3ac8-1264-41b3-b792-d570228a4295 0 drwxr-xr-x. 3 labor labor 55 Oct 8 14:08 1e0894bf-93cf-4796-b2a4-158996457b05 0 drwxr-xr-x. 3 labor labor 55 Oct 8 14:02 1ef837c5-a896-43da-a4a3-15d68644d1ff 0 drwxr-xr-x. 2 labor labor 6 Oct 8 08:54 2387a15c-5353-4de4-bdde-035d9e45ec54 0 drwxr-xr-x. 3 labor labor 59 Oct 9 06:47 38726217-2925-408a-95f6-e65d81524334 0 drwxr-xr-x. 3 labor labor 51 Oct 9 06:23 51f5364b-017c-4ebb-b1d3-d6a0d00df87a 0 drwxr-xr-x. 3 labor labor 63 Oct 8 14:05 65f663f6-413f-4a1e-a20b-fa195ce2351c 0 drwxr-xr-x. 3 labor labor 55 Oct 8 14:00 b00eb3ed-0877-4fc2-aea7-b14e83b556c7 0 drwxr-xr-x. 3 labor labor 55 Oct 8 14:01 ee54738c-b9a7-469b-aaa6-de62cce98598 0 drwxr-xr-x. 2 labor labor 6 Oct 8 09:05 ef0afbac-d5cb-494e-bd18-2c9a7b709427 0 drwxr-xr-x. 3 labor labor 55 Oct 8 13:53 f59236f2-b0da-4617-bfa8-4f4b48016bbe 0 -rw-r--r--. 1 labor labor 0 Oct 9 06:45 test labor@dc19b557640a:/uvd-workflow-orchestrator-example-prefect$
m
thought for 75.6 seconds
Yes — the checkout happens when the runner materializes code for a run, which
serve
triggers. You can test just the checkout step (without serving) by calling the storage’s
pull_code()
— and your results confirm a permission issue on the second pull. What’s going on - First pull creates the destination and copies the repo. - Second pull tries to copy over an existing
.git/objects/pack/*
file tree. Those pack files are often read‑only; combined with an existing tree and/or restrictive directory perms, Python’s copy will hit Errno 13. How to verify just the checkout (no serve) - This is the minimal preflight:
Copy code
from prefect.runner.storage import BlockStorageAdapter
from prefect_gitlab import GitLabRepository, GitLabCredentials
import asyncio

async def main():
    repo = GitLabRepository.load("prefectprototypegit")
    repo.credentials = GitLabCredentials.load("mycredential")

    storage = BlockStorageAdapter(block=repo)
    await storage.pull_code()
    print("Pulled to:", storage.destination)

asyncio.run(main())
If you run that twice and the second run fails with Permission denied in
.git/objects/pack
, it’s definitely a destination reuse/permissions problem. Ways to fix or avoid the error - Fresh destination each time (recommended) - Let Prefect choose a new destination under
${TMPDIR:-/tmp}/runner_storage/<uuid>/...
so you never overwrite an existing tree. - If you previously pointed the storage to a fixed path (e.g., inside your repo), stop overriding the destination and let it be the default. - If you want the runner storage on your mounted workspace, set
TMPDIR
before importing Prefect so the fresh UUID paths land there:
Copy code
export TMPDIR=/workspaces/your-repo/.tmp
    mkdir -p "$TMPDIR" && chmod 777 "$TMPDIR"
- Pre-clean the destination before re-pulling - If you’re intentionally reusing a fixed location:
Copy code
import shutil, pathlib
    if pathlib.Path(storage.destination).exists():
        shutil.rmtree(storage.destination, ignore_errors=True)
    await storage.pull_code()
- Don’t copy .git at all (if your block supports directories) - Limit to only the code dirs you need so
.git
isn’t copied into the destination:
Copy code
repo.directories = ["workflows"]  # adjust to your code dir(s)
- Fix permissions on the existing tree - If you must reuse the same path:
Copy code
chmod -R u+rwX,g+rwX /uvd-workflow-orchestrator-example-prefect/gitlab-repository-prefectprototypegit
    find /uvd-workflow-orchestrator-example-prefect/gitlab-repository-prefectprototypegit/.git/objects/pack -type f -exec chmod u+w,g+w {} +
- Or just remove the existing directory before re-run. Alternative flow (no copy step) - If you prefer to control the checkout yourself and avoid Prefect’s copy entirely: 1) Clone to a clean path you own (fresh each time or pre-clean). 2) Serve from that path.
Copy code
from prefect import flow as flow_cls

  local_path = "/workspaces/your-repo/_src"  # your own clone target
  # ...run `git clone` here, or ensure it's clean...

  f = flow_cls.from_source(
      source=local_path,
      entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
  )
  f.serve(name="private-gitlab-deploy")
Why your /tmp/runner_storage listing looks fine but still fails - The top-level is writable, but your failure path is inside a reused destination that already contains
.git/objects/pack/*
files. Those files are commonly read-only, so a second copy over the same location will fail unless you delete them or ensure the directory strategy creates a fresh destination. References - Storing/serving flow code: Store flow code for deployments - Background on runner storage under tempfile/runner_storage: Related issue and notes If you can share: - The exact
print(storage.destination)
for both runs
-
ls -ld
for that destination and for
.git/objects/pack
I can suggest the minimal change for your setup.
o
@Marvin so you are basically telling me that flow.from_source().serve() is not working? becuase there is dead code lying in /tmp/runner_storage? and i have to clean that up manually?
m
thought for 69.5 seconds
Short answer: No —
flow.from_source(...).serve(...)
works. What you’re hitting is a stale destination with mismatched permissions/ownership, so subsequent pulls try to overwrite
.git/objects/pack/*
and fail with EACCES. That can happen if a prior run created those files with a different user (e.g., root) or with read-only perms. It’s not inherent to serve; it’s the reuse of an existing destination with files you can’t overwrite. Why you see this - The runner clones to a staging tmp dir, then copies into a “runner storage” destination. If that destination (or files within, like
.git/objects/pack/*
) already exist with restrictive perms or different ownership, a second pull/copy fails. - This often occurs if you ran once as root and later as a non-root user, or if you pointed the destination out of
/tmp
to a path owned by a different UID. What to do Pick one of these approaches: 1) One-time cleanup to clear stale, mis-owned files
Copy code
rm -rf ${TMPDIR:-/tmp}/runner_storage/*
# If some dirs are root-owned, run with sudo or fix ownership:
# sudo chown -R "$(id -u):$(id -g)" ${TMPDIR:-/tmp}/runner_storage
Then re-run your script. This usually resolves the immediate error. 2) Ensure a fresh, user-owned temp base and clean at startup Set TMPDIR before importing Prefect so runner storage lands in a path you control, and clean it on startup:
Copy code
import os, shutil
from pathlib import Path

os.environ["TMPDIR"] = "/workspaces/your-repo/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

# Optional: proactively clear prior runner storage for a clean start
shutil.rmtree(Path(os.environ["TMPDIR"]) / "runner_storage", ignore_errors=True)
3) Exclude .git from what’s copied If your GitLab repository block supports selecting directories, only copy your code dirs so
.git
is not included:
Copy code
gitlab_repository_block.directories = ["workflows"]  # adjust to your code dirs
This avoids overwriting .git pack files entirely. 4) Pre-clone yourself and serve from that path If you want full control, clone to a fresh directory you own and serve from there:
Copy code
from prefect import flow as flow_cls

local_path = "/workspaces/your-repo/_src"
# ...git clone or clean this path first...

f = flow_cls.from_source(
    source=local_path,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
How to validate the checkout step without serving You can preflight the clone/pull and see exactly where Prefect writes:
Copy code
from prefect.runner.storage import BlockStorageAdapter
from prefect_gitlab import GitLabRepository, GitLabCredentials
import asyncio

async def main():
    repo = GitLabRepository.load("prefectprototypegit")
    repo.credentials = GitLabCredentials.load("mycredential")
    storage = BlockStorageAdapter(block=repo)
    await storage.pull_code()
    print("Pulled to:", storage.destination)

asyncio.run(main())
Run it twice; if the second run fails, it confirms reuse of an existing destination with files you can’t overwrite. Cleaning or changing TMPDIR should fix it. Next step debug Please share: - Ownership of the failing path (this will confirm a root vs non-root mismatch):
Copy code
ls -l /uvd-workflow-orchestrator-example-prefect/gitlab-repository-prefectprototypegit/.git/objects/pack
- Whether you previously ran the flow as root inside the container. Refs - How Prefect stores/loads code from source: Store flow code for deployments - Related context on runner temp storage: Issue discussion
o
@Marvin ok, so temporarly using codeblock download worked fine. but still if i use from_source().serve() without using the intermediate downloads i get strange issues with shutil errors about downloading hutil.Error: [('/uvd-workflow-orchestrator-example-prefect/.tmp/tmpqc_3r7o2prefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', '/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/44cc86ec-71bb-49a7-93ec-c750bafd242a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack', "[Errno 13] Permission denied: '/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/44cc86ec-71bb-49a7-93ec-c750bafd242a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.pack'"), ('/uvd-workflow-orchestrator-example-prefect/.tmp/tmpqc_3r7o2prefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', '/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/44cc86ec-71bb-49a7-93ec-c750bafd242a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev', "[Errno 13] Permission denied: '/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/44cc86ec-71bb-49a7-93ec-c750bafd242a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.rev'"), ('/uvd-workflow-orchestrator-example-prefect/.tmp/tmpqc_3r7o2prefect/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', '/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/44cc86ec-71bb-49a7-93ec-c750bafd242a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx', "[Errno 13] Permission denied: '/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/44cc86ec-71bb-49a7-93ec-c750bafd242a/gitlab-repository-prefectprototypegit/.git/objects/pack/pack-422b941e3722a2b48413e9f92a217fc0ab1468cb.idx'")]
m
thought for 85.0 seconds
You’ve nailed the root cause: when serve runs, Prefect periodically pulls and then copies the repo from a staging tmp dir into the runner_storage destination. On the second (and later) copies, it tries to overwrite .git/objects/pack/* which are typically read-only; shutil.copy2 fails with EACCES on those files. That’s why your one-off pre-download works, but serve’s periodic pull/copy errors out. You don’t have to abandon flow.from_source(...).serve(...). Pick one of these fixes: Option 1 — Exclude .git by only copying your code dirs If your GitLabRepository block supports directories (like GitRepository does), tell it to copy only your code directory so .git never gets copied:
Copy code
repo = GitLabRepository.load("prefectprototypegit")
repo.credentials = GitLabCredentials.load("mycredential")
repo.directories = ["workflows"]  # adjust to your code dir(s)

f = flow.from_source(
    source=repo,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
This is the cleanest fix because it avoids overwriting .git pack files entirely. Option 2 — Disable periodic pulls for the session If you don’t need auto-refreshing code while serving, disable the pull loop by wrapping your block with BlockStorageAdapter and setting pull_interval=None:
Copy code
from prefect.runner.storage import BlockStorageAdapter

repo = GitLabRepository.load("prefectprototypegit")
repo.credentials = GitLabCredentials.load("mycredential")

storage = BlockStorageAdapter(block=repo, pull_interval=None)  # no periodic re-copy

f = flow.from_source(
    source=storage,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
Option 3 — Pre-clean runner_storage at startup If you want periodic pulls but are hitting stale files, clear the runner destination before serve starts. Set TMPDIR to a path you own and remove the runner_storage dir:
Copy code
import os, shutil, tempfile
from pathlib import Path

os.environ["TMPDIR"] = "/uvd-workflow-orchestrator-example-prefect/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)
shutil.rmtree(Path(tempfile.gettempdir()) / "runner_storage", ignore_errors=True)

# then construct your repo/block and serve as usual
Option 4 — Keep your working pre-download pattern Clone to a fresh directory you own (or clean it first), then serve from that local path:
Copy code
f = flow.from_source(
    source="/uvd-workflow-orchestrator-example-prefect/_src",  # your own clone target
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
Why this happens - Serve uses a fresh UUID destination under ${TMPDIR:-/tmp}/runner_storage. - On each pull_interval (default 60s), it copies again into the same destination. - .git/objects/pack files from the first copy are typically 0444; the second copy tries to overwrite them and gets EACCES. I recommend Option 1 (directories=["workflows"]) if your block supports it; otherwise Option 2 (pull_interval=None) is a quick workaround. If you want, paste your GitLabRepository block’s attribute list and I can confirm the directories setting name for that block. Refs: - Store flow code from source: How-to: store flow code from source - Background on runner storage behavior: Related discussion
o
@Marvin where is .directories defined? i do not see that kind of setter method
m
thought for 122.6 seconds
Great question — and you’re right to look for it. - The directories option exists on Prefect’s core GitRepository (prefect.runner.storage.GitRepository). - It is not available on the prefect-gitlab GitLabRepository block. That block does not expose a directories field, so you can’t exclude .git via that API. Two solid paths forward: A) Switch to core GitRepository (supports directories) - Use your GitLab repo URL and rely on HTTPS with a token (or SSH agent). Then you can copy only your code dir(s) and avoid .git entirely.
Copy code
import os
from prefect import flow as flow_cls
from prefect.runner.storage import GitRepository

# Example using a basic HTTPS URL with a token in env (avoid committing tokens!)
# For GitLab, username can be anything; many use 'oauth2' or your actual username.
GITLAB_TOKEN = os.environ["GITLAB_TOKEN"]
url = f"<https://oauth2:{GITLAB_TOKEN}@gitlab.com/your-group/your-repo.git>"

repo = GitRepository(
    url=url,
    directories=["workflows"],   # only copy your flow code; .git is not copied
    pull_interval=60,            # or None to disable periodic pulls
)

f = flow_cls.from_source(
    source=repo,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
- This avoids the shutil EACCES on .git/objects/pack during the periodic copy. B) Keep GitLabRepository but avoid periodic re-copying - Wrap your block with BlockStorageAdapter and disable the periodic pull that triggers the overwrite of .git pack files:
Copy code
from prefect.runner.storage import BlockStorageAdapter
from prefect_gitlab import GitLabRepository, GitLabCredentials
from prefect import flow as flow_cls

repo = GitLabRepository.load("prefectprototypegit")
repo.credentials = GitLabCredentials.load("mycredential")

storage = BlockStorageAdapter(block=repo, pull_interval=None)  # no periodic re-copy

f = flow_cls.from_source(
    source=storage,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
- This keeps the GitLab block but prevents the second copy (which is when .git overwrite fails). If you still want periodic updates, you can implement a manual refresh step between restarts or set a long pull_interval to reduce churn. Why directories matters here - The failure occurs when serve copies from a staging clone to the runner_storage destination. On the second copy, .git/objects/pack files from the first copy are read-only and can’t be overwritten. Excluding .git avoids this entirely, but that capability is only present on core GitRepository. Useful links - Store flow code from source: How-to: store flow code from source - prefect-gitlab integration overview: prefect-gitlab integration If you want, paste your GitLab URL pattern (SSH vs HTTPS), and I’ll tailor the GitRepository example with the right auth pattern for your setup.
o
@Marvin when i use B it isencountering an value error n self._loop.run_until_complete(task) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete | return future.result() | ~~~~~~~~~~~~~^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 506, in start | async with self as runner: | ^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 1589, in aexit | await self._exit_stack.__aexit__(*exc_info) | File "/usr/local/lib/python3.13/contextlib.py", line 768, in aexit | raise exc | File "/usr/local/lib/python3.13/contextlib.py", line 751, in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in aexit | raise BaseExceptionGroup( | "unhandled errors in a TaskGroup", self._exceptions | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 509, in start | async with self._loops_task_group as loops_task_group: | ^^^^^^^^^^^^^^^^^^^^^^ | File "/home/labor/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in aexit | raise BaseExceptionGroup( | "unhandled errors in a TaskGroup", self._exceptions | ) from None | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/labor/.local/lib/python3.13/site-packages/prefect/utilities/services.py", line 64, in critical_service_loop | await workload() | File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/storage.py", line 283, in pull_code | raise ValueError( | ...<2 lines>... | ) | ValueError: The existing repository at /uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/b2f7289f-38dc-47a8-9fb4-ac790676347d/uvd-workflow-orchestrator-example-prefect.git does not match the configured repository
m
thought for 38.6 seconds
That ValueError means Prefect found an existing Git repo at the destination and its remote does not match the repo configured on your GitLabRepository block. Prefect refuses to overwrite it, so it raises. Why this happens - The runner creates a destination under ${TMPDIR}/runner_storage/<uuid>/<repo-name>.git - If that folder already contains a Git repo (e.g., from a prior run with a different remote), Prefect detects the mismatch and errors out. Quick fix right now - Remove the specific folder from the error message and try again:
Copy code
rm -rf "/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/b2f7289f-38dc-47a8-9fb4-ac790676347d/uvd-workflow-orchestrator-example-prefect.git"
- Or clean all runner storage (safe in dev):
Copy code
rm -rf "/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage"
Make it robust at startup - Set TMPDIR and pre-clean runner_storage before importing Prefect, so you always start fresh:
Copy code
import os, shutil, tempfile
from pathlib import Path

os.environ["TMPDIR"] = "/uvd-workflow-orchestrator-example-prefect/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

# Clear any stale repos to avoid remote mismatch / EACCES on .git/objects/pack
shutil.rmtree(Path(tempfile.gettempdir()) / "runner_storage", ignore_errors=True)
Then keep your Option B pattern:
Copy code
from prefect.runner.storage import BlockStorageAdapter
from prefect_gitlab import GitLabRepository, GitLabCredentials
from prefect import flow as flow_cls

repo = GitLabRepository.load("prefectprototypegit")
repo.credentials = GitLabCredentials.load("mycredential")

storage = BlockStorageAdapter(block=repo, pull_interval=None)  # no periodic re-copy

f = flow_cls.from_source(
    source=storage,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
If you want to diagnose the mismatch before deleting, inspect the existing remote:
Copy code
git -C "/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage/b2f7289f-38dc-47a8-9fb4-ac790676347d/uvd-workflow-orchestrator-example-prefect.git" remote -v
Compare with your GitLabRepository’s configured URL. Alternative that avoids these .git issues entirely - Use the core GitRepository (supports directories) and only copy your code directory (not .git). This avoids both read-only pack files and remote mismatches:
Copy code
import os
from prefect import flow as flow_cls
from prefect.runner.storage import GitRepository

os.environ["TMPDIR"] = "/uvd-workflow-orchestrator-example-prefect/.tmp"

# Use HTTPS with a token in env for GitLab
url = f"<https://oauth2:{os.environ['GITLAB_TOKEN']}@gitlab.com/your-group/your-repo.git>"

repo = GitRepository(
    url=url,
    directories=["workflows"],   # only copy your flow code; .git is not copied
    pull_interval=60,            # or None if you don't want periodic refresh
)

f = flow_cls.from_source(
    source=repo,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
Two final notes - You’re on Python 3.13; Prefect 3.x is tested on 3.10–3.12. If you still see odd asyncio/serve behavior after cleaning runner_storage, try 3.12. - Docs for storing code from source: Store flow code from source If you share the output of the git remote -v command for that path, I can confirm whether it’s a real mismatch or just a stale folder collision.
o
@Marvin when i use blockstorageadapter it serves but fails at run do = await self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 153, in iter result = await action(retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/tenacity/_utils.py", line 99, in inner return call(*args, **kwargs) File "/home/labor/.local/lib/python3.13/site-packages/tenacity/__init__.py", line 420, in exc_check raise retry_exc.reraise() ~~~~~~~~~~~~~~~~~^^ File "/home/labor/.local/lib/python3.13/site-packages/tenacity/__init__.py", line 187, in reraise raise self.last_attempt.result() ~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/labor/.local/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 114, in call result = await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/prefect_gitlab/repositories.py", line 161, in get_directory cmd = ["git", "clone", self._create_repo_url()] ~~~~~~~~~~~~~~~~~~~~~^^ File "/home/labor/.local/lib/python3.13/site-packages/prefect_gitlab/repositories.py", line 108, in _create_repo_url token = self.credentials.token.get_secret_value() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get_secret_value' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/labor/.local/lib/python3.13/site-packages/prefect/runner/runner.py", line 1527, in _run_on_crashed_hooks flow = await load_flow_from_flow_run( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ flow_run, storage_base_path=str(self._tmp_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/labor/.local/lib/python3.13/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/labor/.local/lib/python3.13/site-packages/prefect/flows.py", line 2810, in load_flow_from_flow_run output = await run_steps( ^^^^^^^^^^^^^^^^ deployment.pull_steps, print_function=run_logger.info ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/labor/.local/lib/python3.13/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps raise StepExecutionError(f"Encountered error while running {fqn}") from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.pull_with_block
m
thought for 34.3 seconds
Thanks for the stack — that error is coming from prefect-gitlab because the GitLabRepository block that’s being used at RUN TIME has no credentials attached: AttributeError: 'NoneType' object has no attribute 'get_secret_value' → self.credentials is None inside the subprocess that pulls code. Why this happens - When you call serve, Prefect creates a deployment and runs flow runs in a child process. - That child process re-loads the repository block from the Prefect API (via the pull_with_block step). - Your line
repo.credentials = GitLabCredentials.load("mycredential")
only mutates the in-memory object. If you don’t save it back, the block in the API still has no credentials, so the subprocess sees
credentials=None
and fails. Fix: persist the credentials on the block before serving Do this once, then serve. Make sure you do this before importing/using the block for from_source.
Copy code
from prefect_gitlab import GitLabRepository, GitLabCredentials

# Ensure your credentials block exists and has the token
creds = GitLabCredentials.load("mycredential")  # or create and save it if needed

# Attach creds to the repo block and SAVE it so the API has them
repo = GitLabRepository.load("prefectprototypegit")
repo.credentials = creds
repo.save("prefectprototypegit", overwrite=True)

# Sanity check: new load should show credentials present
assert GitLabRepository.load("prefectprototypegit").credentials is not None
Then use the saved block with BlockStorageAdapter (to avoid the .git pack overwrite) or use it directly:
Copy code
import os
from pathlib import Path
from prefect import flow as flow_cls
from prefect.runner.storage import BlockStorageAdapter
from prefect_gitlab import GitLabRepository

# Optional but recommended: control temp base and start clean
os.environ["TMPDIR"] = "/uvd-workflow-orchestrator-example-prefect/.tmp"
Path(os.environ["TMPDIR"]).mkdir(parents=True, exist_ok=True)

repo = GitLabRepository.load("prefectprototypegit")

# Disable periodic recopy to avoid .git pack overwrites
storage = BlockStorageAdapter(block=repo, pull_interval=None)

f = flow_cls.from_source(
    source=storage,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
Alternative: use core GitRepository (supports directories) If you’d rather avoid copying .git entirely, switch to Prefect’s core GitRepository and copy only your code directory:
Copy code
from prefect.runner.storage import GitRepository
from prefect import flow as flow_cls
import os

os.environ["TMPDIR"] = "/uvd-workflow-orchestrator-example-prefect/.tmp"

url = f"<https://oauth2:{os.environ['GITLAB_TOKEN']}@gitlab.com/your-group/your-repo.git>"

repo = GitRepository(
    url=url,
    directories=["workflows"],   # only copy your flow code; .git not copied
    pull_interval=60,            # or None to disable periodic refresh
)

f = flow_cls.from_source(
    source=repo,
    entrypoint="workflows/workflow_with_non_and_decorated_task.py:my_flow",
)
f.serve(name="private-gitlab-deploy")
Two additional tips - Clean runner storage between experiments to avoid stale repos/permissions:
Copy code
rm -rf "/uvd-workflow-orchestrator-example-prefect/.tmp/runner_storage"
- You’re on Python 3.13; Prefect 3.x is tested on 3.10–3.12. If you see unrelated asyncio issues, try 3.12 to rule that out. Docs - GitLab integration overview: prefect-gitlab - Storing/serving flow code from source: Store flow code from source If you run the assert check and it still shows credentials missing, paste the output of:
Copy code
from prefect_gitlab import GitLabRepository
r = GitLabRepository.load("prefectprototypegit")
print("Has creds?", r.credentials is not None)
and we’ll sort out the block contents.