https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> could you help with this? i am trying...
# ask-marvin
a
@Marvin could you help with this? i am trying to deploy my project, i am using a gitlab-credentail block for that and all of my colleagues could run the deployment using same block but mine is not working and i get this error when i run it?
Copy code
Worker 'KubernetesWorker c471c70f-6c61-4456-b125-4242350cac2f' submitting flow run '0fc3493f-c567-479c-905b-6e8c5db20f46'
04:11:25 PM
prefect.flow_runs.worker

Creating Kubernetes job...
04:11:25 PM
prefect.flow_runs.worker

Completed submission of flow run '0fc3493f-c567-479c-905b-6e8c5db20f46'
04:11:25 PM
prefect.flow_runs.worker

Opening process...
04:11:37 PM
prefect.flow_runs.runner

 > Running git_clone step...
04:11:39 PM
prefect.flow_runs

Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.
04:11:43 PM
prefect.flow_runs

Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 204, in git_clone
    run_coro_as_sync(_pull_git_repository_with_retries(storage))
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 197, in run_coro_as_sync
    result = from_sync.call_in_new_thread(coroutine_wrapper)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/api.py", line 220, in call_in_new_thread
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/retries.py", line 57, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 47, in _pull_git_repository_with_retries
    await repo.pull_code()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 415, in pull_code
    await self._clone_repo()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 460, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/engine.py", line 112, in <module>
    flow: "Flow[..., Any]" = load_flow(flow_run)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/flow_engine.py", line 150, in load_flow
    flow = run_coro_as_sync(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run
    output = await run_steps(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
04:11:43 PM
prefect.flow_runs

Process for flow run 'holistic-ladybug' exited with status code: 1
04:11:43 PM
prefect.flow_runs.runner

Reported flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' as crashed: Flow run process exited with non-zero status code 1.
04:11:43 PM
prefect.flow_runs.runner

Loading flow to check for on_crashed hooks
04:11:43 PM
prefect.flow_runs.runner

 > Running git_clone step...
04:11:43 PM
prefect.flow_runs

Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.
04:11:47 PM
prefect.flow_runs

Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('0fc3493f-c567-479c-905b-6e8c5db20f46').
Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 204, in git_clone
    run_coro_as_sync(_pull_git_repository_with_retries(storage))
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/retries.py", line 57, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 47, in _pull_git_repository_with_retries
    await repo.pull_code()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 415, in pull_code
    await self._clone_repo()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 460, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/runner.py", line 1548, in _run_on_crashed_hooks
    flow = await load_flow_from_flow_run(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run
    output = await run_steps(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
04:11:47 PM
prefect.flow_runs.runner
and here is my code from prefect import flow from prefect.runner.storage import GitRepository from prefect_gitlab import GitLabCredentials if name == "__main__": gitlab_repo = GitRepository( url="https://////.git", credentials=GitLabCredentials.load("gitlab-creds"), branch="dev" ) flow.from_source( source=gitlab_repo, entrypoint="workflow.py:norway_pkt", ).deploy( name="norway-pkt-deployment", work_pool_name="kubernetes", job_variables={ "image_pull_policy": "Always", "memory_request": "32Gi", }, parameters={"snapshot_date": "current"}, image="capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1", build=False, push=False, )
m
Your message was too long, here's your message at the allowed limit: could you help with this? i am trying to deploy my project i am using a gitlab-credentail block for that and all of my colleagues could run the deployment using same block but mine is not working and i get this error when i run it? ```Worker 'KubernetesWorker c471c70f-6c61-4456-b125-4242350cac2f' submitting flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' 04 11 25 PM prefect flow_runs worker Creating Kubernetes job 04 11 25 PM prefect flow_runs worker Completed submission of flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' 04 11 25 PM prefect flow_runs worker Opening process 04 11 37 PM prefect flow_runs runner > Running git_clone step 04 11 39 PM prefect flow_runs Failed to clone repository '<https //gitlab devops prdroot net/service/dwh/data-science-no/production-models/norway-pkt git>' with exit code 128 04 11 43 PM prefect flow_runs Unexpected exception encountered when trying to load flow Traceback (most recent call last) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/core py line 166 in run_steps step_output = await run_step(step upstream_outputs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/core py line 137 in run_step result = await from_async call_soon_in_new_thread( File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 374 in aresult return await asyncio wrap_future(self future) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 399 in _run_sync result = self fn(*self args *self kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/compatibility/async_dispatch py line 94 in wrapper return fn(*args *kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/pull py line 204 in git_clone run_coro_as_sync(_pull_git_repository_with_retries(storage)) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 197 in run_coro_as_sync result = from_sync call_in_new_thread(coroutine_wrapper) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/api py line 220 in call_in_new_thread return call result() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 365 in result return self future result(timeout=timeout) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 192 in result return self __get_result() File /usr/local/lib/python3 10/concurrent/futures/_base py line 403 in __get_result raise self _exception File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 441 in _run_async result = await coro File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 188 in coroutine_wrapper return await task File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/retries py line 57 in wrapper return await func(*args *kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/pull py line 47 in _pull_git_repository_with_retries await repo pull_code() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/runner/storage py line 415 in pull_code await self _clone_repo() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/runner/storage py line 460 in _clone_repo raise RuntimeError( RuntimeError Failed to clone repository '<https //gitlab devops prdroot net/service/dwh/data-science-no/production-models/norway-pkt git>' with exit code 128 The above exception was the direct cause of the following exception Traceback (most recent call last) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/engine py line 112 in <module> flow Flow[ Any] = load_flow(flow_run) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/flow_engine py line 150 in load_flow flow = run_coro_as_sync( File /opt/venv_prefe
ct/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 207 in run_coro_as_sync return call result() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 365 in result return self future result(timeout=timeout) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 192 in result return self __get_result() File /usr/local/lib/python3 10/concurrent/futures/_base py line 403 in __get_result raise self _exception File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 441 in _run_async result = await coro File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 188 in coroutine_wrapper return await task File /opt/venv_prefect/lib/python3 10/site-packages/prefect/client/utilities py line 69 in wrapper return await func(client *args *kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/flows py line 2816 in load_flow_from_flow_run output = await run_steps( File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/core py line 194 in run_steps raise StepExecutionError(f Encountered error while running {fqn} ) from exc prefect
a
ok here is my code
Copy code
if __name__ == "__main__":
    gitlab_repo = GitRepository(
        url="https://////.git",
        credentials=GitLabCredentials.load("gitlab-creds"),
        branch="dev"
    )

    flow.from_source(
        source=gitlab_repo,
        entrypoint="workflow.py:norway_pkt",
    ).deploy(
        name="norway-pkt-deployment",
        work_pool_name="kubernetes",
        job_variables={
            "image_pull_policy": "Always",
            "memory_request": "32Gi",
        },
        parameters={"snapshot_date": "current"},
        image="<http://capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1|capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1>",
        build=False,
        push=False,
    )
ok here is my code @Marvin
Copy code
if __name__ == "__main__":
    gitlab_repo = GitRepository(
        url="https://////.git",
        credentials=GitLabCredentials.load("gitlab-creds"),
        branch="dev"
    )

    flow.from_source(
        source=gitlab_repo,
        entrypoint="workflow.py:norway_pkt",
    ).deploy(
        name="norway-pkt-deployment",
        work_pool_name="kubernetes",
        job_variables={
            "image_pull_policy": "Always",
            "memory_request": "32Gi",
        },
        parameters={"snapshot_date": "current"},
        image="capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1",
        build=False,
        push=False,
    )
m
✅ thought for 447.4 seconds
Thanks for sharing your snippet! A few quick fixes and notes to make this work as intended: 1) Use Flow.from_source (or your decorated flow object), not the decorator symbol flow - Calling 
flow.from_source(...)
will refer to the decorator function from Prefect, not your flow. - Either import and use 
Flow.from_source(...)
, or call 
norway_pkt.from_source(...)
if you’ve defined/imported the decorated flow function locally. 2) Kubernetes job_variables - 
image_pull_policy
is a valid top-level key and will work as-is. - 
memory_request
(and CPU/memory limits) will not do anything unless your Kubernetes work pool’s Base Job Template defines those variables and references them in the pod spec. If you want to set resources, add variables like 
memory_request
, 
memory_limit
, 
cpu_request
, 
cpu_limit
to the pool’s template and wire them into 
.spec.template.spec.containers[0].resources
. Example snippet to add to your work pool’s Base Job Template (Variables + usage):
Copy code
{
  "variables": {
    "cpu_request": {
      "type": "string",
      "default": "250m"
    },
    "cpu_limit": {
      "type": "string",
      "default": "500m"
    },
    "memory_request": {
      "type": "string",
      "default": "512Mi"
    },
    "memory_limit": {
      "type": "string",
      "default": "1Gi"
    }
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "resources": {
                  "requests": {
                    "cpu": "{{ cpu_request }}",
                    "memory": "{{ memory_request }}"
                  },
                  "limits": {
                    "cpu": "{{ cpu_limit }}",
                    "memory": "{{ memory_limit }}"
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}
3) GitLab credentials - Ensure you’ve installed and registered the GitLab block type before calling `GitLabCredentials.load("gitlab-creds")`:
Copy code
pip install -U prefect-gitlab
prefect block register -m prefect_gitlab
Here’s a corrected version of your script with imports and the right 
from_source
call:
Copy code
from prefect.flows import Flow
from prefect.runner.storage import GitRepository
from prefect_gitlab import GitLabCredentials

if __name__ == "__main__":
    gitlab_repo = GitRepository(
        url="<https://gitlab.your-host.com/group/project.git>",
        credentials=GitLabCredentials.load("gitlab-creds"),
        branch="dev",
    )

    Flow.from_source(
        source=gitlab_repo,
        entrypoint="workflow.py:norway_pkt",
    ).deploy(
        name="norway-pkt-deployment",
        work_pool_name="kubernetes",
        image="<http://capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1|capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1>",
        build=False,
        push=False,
        parameters={"snapshot_date": "current"},
        job_variables={
            "image_pull_policy": "Always",
            # The following only take effect if your pool template defines/uses them:
            "memory_request": "32Gi",
            # e.g., also consider "cpu_request", "memory_limit", "cpu_limit"
        },
    )
Questions to help me tailor next steps: - Are you seeing an error, or is 
memory_request
just not being applied on the pod? - Do you want help updating your Kubernetes work pool’s Base Job Template to add CPU/memory requests/limits (and optionally env, labels/annotations)?
a
i get this error @Marvin
Copy code
Worker 'KubernetesWorker c471c70f-6c61-4456-b125-4242350cac2f' submitting flow run '0fc3493f-c567-479c-905b-6e8c5db20f46'
04:11:25 PM
prefect.flow_runs.worker

Creating Kubernetes job...
04:11:25 PM
prefect.flow_runs.worker

Completed submission of flow run '0fc3493f-c567-479c-905b-6e8c5db20f46'
04:11:25 PM
prefect.flow_runs.worker

Opening process...
04:11:37 PM
prefect.flow_runs.runner

 > Running git_clone step...
04:11:39 PM
prefect.flow_runs

Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.
04:11:43 PM
prefect.flow_runs

Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 204, in git_clone
    run_coro_as_sync(_pull_git_repository_with_retries(storage))
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 197, in run_coro_as_sync
    result = from_sync.call_in_new_thread(coroutine_wrapper)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/api.py", line 220, in call_in_new_thread
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/retries.py", line 57, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 47, in _pull_git_repository_with_retries
    await repo.pull_code()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 415, in pull_code
    await self._clone_repo()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 460, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/engine.py", line 112, in <module>
    flow: "Flow[..., Any]" = load_flow(flow_run)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/flow_engine.py", line 150, in load_flow
    flow = run_coro_as_sync(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run
    output = await run_steps(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
04:11:43 PM
prefect.flow_runs

Process for flow run 'holistic-ladybug' exited with status code: 1
04:11:43 PM
prefect.flow_runs.runner

Reported flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' as crashed: Flow run process exited with non-zero status code 1.
04:11:43 PM
prefect.flow_runs.runner

Loading flow to check for on_crashed hooks
04:11:43 PM
prefect.flow_runs.runner

 > Running git_clone step...
04:11:43 PM
prefect.flow_runs

Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.
04:11:47 PM
prefect.flow_runs

Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('0fc3493f-c567-479c-905b-6e8c5db20f46').
Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 204, in git_clone
    run_coro_as_sync(_pull_git_repository_with_retries(storage))
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/retries.py", line 57, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 47, in _pull_git_repository_with_retries
    await repo.pull_code()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 415, in pull_code
    await self._clone_repo()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/storage.py", line 460, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/runner/runner.py", line 1548, in _run_on_crashed_hooks
    flow = await load_flow_from_flow_run(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/flows.py", line 2816, in load_flow_from_flow_run
    output = await run_steps(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
04:11:47 PM
prefect.flow_runs.runner
m
Your message was too long, here's your message at the allowed limit: i get this error ```Worker 'KubernetesWorker c471c70f-6c61-4456-b125-4242350cac2f' submitting flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' 04 11 25 PM prefect flow_runs worker Creating Kubernetes job 04 11 25 PM prefect flow_runs worker Completed submission of flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' 04 11 25 PM prefect flow_runs worker Opening process 04 11 37 PM prefect flow_runs runner > Running git_clone step 04 11 39 PM prefect flow_runs Failed to clone repository '<https //gitlab devops prdroot net/service/dwh/data-science-no/production-models/norway-pkt git>' with exit code 128 04 11 43 PM prefect flow_runs Unexpected exception encountered when trying to load flow Traceback (most recent call last) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/core py line 166 in run_steps step_output = await run_step(step upstream_outputs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/core py line 137 in run_step result = await from_async call_soon_in_new_thread( File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 374 in aresult return await asyncio wrap_future(self future) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 399 in _run_sync result = self fn(*self args *self kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/compatibility/async_dispatch py line 94 in wrapper return fn(*args *kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/pull py line 204 in git_clone run_coro_as_sync(_pull_git_repository_with_retries(storage)) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 197 in run_coro_as_sync result = from_sync call_in_new_thread(coroutine_wrapper) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/api py line 220 in call_in_new_thread return call result() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 365 in result return self future result(timeout=timeout) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 192 in result return self __get_result() File /usr/local/lib/python3 10/concurrent/futures/_base py line 403 in __get_result raise self _exception File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 441 in _run_async result = await coro File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 188 in coroutine_wrapper return await task File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/retries py line 57 in wrapper return await func(*args *kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/pull py line 47 in _pull_git_repository_with_retries await repo pull_code() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/runner/storage py line 415 in pull_code await self _clone_repo() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/runner/storage py line 460 in _clone_repo raise RuntimeError( RuntimeError Failed to clone repository '<https //gitlab devops prdroot net/service/dwh/data-science-no/production-models/norway-pkt git>' with exit code 128 The above exception was the direct cause of the following exception Traceback (most recent call last) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/engine py line 112 in <module> flow Flow[ Any] = load_flow(flow_run) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/flow_engine py line 150 in load_flow flow = run_coro_as_sync( File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 207 in run_coro_as_sync return call result() File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 365 i
n result return self future result(timeout=timeout) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 192 in result return self __get_result() File /usr/local/lib/python3 10/concurrent/futures/_base py line 403 in __get_result raise self _exception File /opt/venv_prefect/lib/python3 10/site-packages/prefect/_internal/concurrency/calls py line 441 in _run_async result = await coro File /opt/venv_prefect/lib/python3 10/site-packages/prefect/utilities/asyncutils py line 188 in coroutine_wrapper return await task File /opt/venv_prefect/lib/python3 10/site-packages/prefect/client/utilities py line 69 in wrapper return await func(client *args *kwargs) File /opt/venv_prefect/lib/python3 10/site-packages/prefect/flows py line 2816 in load_flow_from_flow_run output = await run_steps( File /opt/venv_prefect/lib/python3 10/site-packages/prefect/deployments/steps/core py line 194 in run_steps raise StepExecutionError(f Encountered error while running {fqn} ) from exc prefect deployments steps core StepExecutionError Encountered error while running prefect deployments steps git_clone 04 11 43 PM prefect flow_runs Process for flow run 'holistic-ladybug' exited with status code 1 04 11 43 PM prefect flow_runs runner Reported flow run '0fc3493f-c567-479c-905b-6e8c5db20f46' as crashed Flow
a
ok here is my error @Marvin
Copy code
Worker 'KubernetesWorker c471c70f-6c61-4456-b125-4242350cac2f' submitting flow run '0fc3493f-c567-479c-905b-6e8c5db20f46'
04:11:25 PM
prefect.flow_runs.worker

Creating Kubernetes job...
04:11:25 PM
prefect.flow_runs.worker

Completed submission of flow run '0fc3493f-c567-479c-905b-6e8c5db20f46'
04:11:25 PM
prefect.flow_runs.worker

Opening process...
04:11:37 PM
prefect.flow_runs.runner

 > Running git_clone step...
04:11:39 PM
prefect.flow_runs

Failed to clone repository '<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>' with exit code 128.
04:11:43 PM
prefect.flow_runs

Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 137, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper
    return fn(*args, **kwargs)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 204, in git_clone
    run_coro_as_sync(_pull_git_repository_with_retries(storage))
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 197, in run_coro_as_sync
    result = from_sync.call_in_new_thread(coroutine_wrapper)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/api.py", line 220, in call_in_new_thread
    return call.result()
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async
    result = await coro
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
  File "/opt/venv_prefect/lib/python3.10/site-packages/prefect/_internal/retries.py", line 57, in wrapper
m
✅ thought for 289.4 seconds
Thanks for the logs — that “Failed to clone repository … with exit code 128” during the git_clone step typically comes from one of these causes (ordered by likelihood given your setup): 1) The runtime image does not have git installed - You’re using a custom image (`capcr.azurecr.io/dsno/norway-pkt-combined:1.0.1`) with build=False/push=False, so Prefect won’t add git for you. - The git_clone step shells out to the git binary. If it’s missing (or too old), you’ll see exit code 128. Quick check inside the running pod:
Copy code
which git && git --version
If it’s missing, rebuild your image with git installed (and CA certs if you’re on a slim base):
Copy code
RUN apt-get update && apt-get install -y --no-install-recommends git ca-certificates && rm -rf /var/lib/apt/lists/*
2) Credentials are not being applied at runtime - If the runtime environment cannot deserialize/use your 
GitLabCredentials
block, the clone will be attempted without auth and fail with 128 (“Authentication failed”). Two ways to fix: - Ensure the runtime image includes the integration that defines your block:
Copy code
pip install -U prefect-gitlab
  prefect block register -m prefect_gitlab
- Or avoid needing the integration package at runtime by passing a plain token dict (core Prefect handles GitLab token formatting):
Copy code
from prefect.runner.storage import GitRepository

  gitlab_repo = GitRepository(
      url="<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>",
      credentials={"access_token": "<your-token-or-secret>"},
      branch="dev",
  )
If you prefer a Secret block:
Copy code
from prefect.blocks.system import Secret
  token = Secret.load("gitlab-token").get()  # your saved PAT
  gitlab_repo = GitRepository(
      url="<https://gitlab.devops.prdroot.net/.../norway-pkt.git>",
      credentials={"access_token": token},
      branch="dev",
  )
3) TLS/Network to your self‑hosted GitLab - For internal GitLab with a private CA, the container may fail SSL verification. Make sure your CA chain is present in the image and updated:
Copy code
COPY your-root-ca.crt /usr/local/share/ca-certificates/
  RUN update-ca-certificates
- Also verify basic connectivity from the pod:
Copy code
curl -I <https://gitlab.devops.prdroot.net>
Next best step to pinpoint: - Can you exec into a failing pod and run a manual clone?
Copy code
git clone <https://oauth2:<PAT>@gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>
If that works, credentials and CA are fine — focus on making sure Prefect passes them (point 2). If that fails, it’ll print the exact auth/TLS error. Optional: turn on DEBUG logs to see the git hint Prefect extracts - Set 
PREFECT_LOGGING_LEVEL=DEBUG
in your runtime and re-run; Prefect usually appends a helpful hint (e.g., “Authentication failed”, “Could not resolve host”, “SSL certificate problem”). If you share: - Whether 
git --version
exists in your image - Whether the manual 
git clone
inside the pod succeeds - Whether you’re installing 
prefect-gitlab
in the runtime image …I can give you a precise fix. My hunch: your custom image is missing git and/or the CA bundle; second hunch is the credentials formatter not available at runtime unless you install 
prefect-gitlab
(using the plain 
credentials={"access_token": ...}
sidesteps that).
a
i can show my image if you see any problem there @Marvin
Copy code
# syntax=docker/dockerfile:1

# -----------python3.10---------------
FROM python:3.10-bookworm AS py310
WORKDIR /build

RUN python -m venv /opt/venv_prefect
COPY requirements_prefect.txt /build/requirements_prefect.txt
RUN /opt/venv_prefect/bin/pip install --no-cache-dir -U pip \
 && /opt/venv_prefect/bin/pip install --no-cache-dir -r /build/requirements_prefect.txt

RUN python -m venv /opt/venv_dbt
COPY requirements.txt /build/requirements.txt
RUN /opt/venv_dbt/bin/pip install --no-cache-dir -U pip \
 && /opt/venv_dbt/bin/pip install --no-cache-dir -r /build/requirements.txt

#-------------------python3.7------------------
FROM python:3.7 AS py37
WORKDIR /build

COPY py_models/requirements.txt /build/requirements_py37.txt
RUN pip install --no-cache-dir -U pip \
 && pip install --no-cache-dir -r /build/requirements_py37.txt

#-------------------final runtime--------------------
FROM python:3.10-bookworm AS final

ENV DEBIAN_FRONTEND=noninteractive \
    ORACLE_IC_HOME="/opt/oracle/instantclient_21_1" \
    LD_LIBRARY_PATH="/opt/oracle/instantclient_21_1" \
    PREFECT_PY="/opt/venv_prefect/bin/python" \
    DBT_BIN="/opt/venv_dbt/bin/dbt" \
    PY37_BIN="/opt/python37/bin/python" \
    PATH="/opt/venv_prefect/bin:/opt/venv_dbt/bin:/opt/python37/bin:${PATH}"

RUN apt-get update -qq && apt-get install -y --no-install-recommends \
    ca-certificates wget unzip libaio1 \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /opt/oracle && cd /opt/oracle \
 && wget -q -c <https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-basic-linux.x64-21.1.0.0.0.zip> \
 && unzip -q instantclient-basic-linux.x64-21.1.0.0.0.zip \
 && echo "${ORACLE_IC_HOME}" > /etc/ld.so.conf.d/oracle-instantclient.conf \
 && ldconfig \
 && rm -f instantclient-basic-linux.x64-21.1.0.0.0.zip

COPY --from=py310 /opt/venv_prefect /opt/venv_prefect
COPY --from=py310 /opt/venv_dbt     /opt/venv_dbt
COPY --from=py37 /usr/local /opt/python37

WORKDIR /app
COPY . /app

RUN mkdir -p /root/.dbt

CMD ["bash"]
m
✅ thought for 43.0 seconds
Thanks for sharing your Dockerfile — I see the likely root cause: - Your final image does not install git. The git_clone step shells out to the git binary; without it, clones fail (often surfacing as exit code 128). What to change in your image - Install git (and optionally openssh-client if you ever use SSH URLs; git‑lfs if your repo uses LFS). - If your GitLab uses a private CA, add that CA to the trust store. Minimal change to your existing apt-get line:
Copy code
RUN apt-get update -qq && apt-get install -y --no-install-recommends \
    ca-certificates wget unzip libaio1 git openssh-client \
 && rm -rf /var/lib/apt/lists/*
If you have a private CA:
Copy code
COPY rootCA.crt /usr/local/share/ca-certificates/rootCA.crt
RUN update-ca-certificates
If your repo uses Git LFS:
Copy code
RUN apt-get update -qq && apt-get install -y --no-install-recommends git-lfs && git lfs install && rm -rf /var/lib/apt/lists/*
Credentials note - If you’re using 
GitLabCredentials.load("gitlab-creds")
, ensure the runtime image can handle that (i.e., 
prefect-gitlab
is installed in 
/opt/venv_prefect
). Easiest is to include it in 
requirements_prefect.txt
. - Alternatively, avoid the integration at runtime by passing a plain token dict to `GitRepository`:
Copy code
gitlab_repo = GitRepository(
    url="<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>",
    credentials={"access_token": "<your-token>"},
    branch="dev",
)
Quick verification inside a running pod - After rebuilding and redeploying, exec into the pod and confirm:
Copy code
which git && git --version
git clone <https://oauth2:<PAT>@gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git> /tmp/clone-test
- If the manual clone succeeds but Prefect still fails, we’ll focus on how credentials are being passed. If it fails with SSL errors, add your CA as shown above. Want me to suggest a minimal, full Dockerfile diff including these changes based on your current file?
a
it has git insatlled and also prefect-gitlab @Marvin
m
✅ thought for 196.1 seconds
Got it — if git and prefect-gitlab are present in the image, the remaining common causes for exit code 128 in the git_clone step are: Most likely culprits - TLS/CA trust for your internal GitLab - If your GitLab uses a private/enterprise CA, the container must trust it. Otherwise you’ll get 128 with “SSL certificate problem”. - Auth/permissions - The PAT in your 
gitlab-creds
block must have read_repository on that project and be accessible to the worker’s workspace at runtime. - The branch 
dev
must exist and be readable by that token. Quick, targeted diagnostics inside the running pod Please exec into the flow-run pod and run these: - Confirm git and version
Copy code
which git && git --version
- Verify Prefect can load your block and the integration is importable
Copy code
/opt/venv_prefect/bin/python - << 'PY'
from prefect_gitlab import GitLabCredentials
print("prefect-gitlab import: OK")
PY
- Confirm the pod has Prefect API env so it can resolve block values at runtime
Copy code
env | grep PREFECT_API_
You should see 
PREFECT_API_URL
and 
PREFECT_API_KEY
. If they’re missing, the block reference won’t be resolvable in the job. - Test network and TLS to GitLab
Copy code
curl -vI <https://gitlab.devops.prdroot.net>
If you see certificate errors, you’ll need to add your org’s root CA into the image:
Copy code
COPY rootCA.crt /usr/local/share/ca-certificates/rootCA.crt
RUN update-ca-certificates
- Manual clone with token to expose the real error Replace <PAT> with a real token that should have repo access:
Copy code
git ls-remote <https://oauth2:<PAT>@gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>
- If this fails with “SSL certificate problem” → add CA as above. - If it fails with “Authentication failed” or 403 → check token scope/access and that the repo/branch exists. - If it fails with “Remote branch dev not found” → verify the 
dev
branch. Turn on Prefect DEBUG logs for more hinting Set this env on your work pool/pod and re-run to see the precise git stderr hint that Prefect appends:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
A couple of fast mitigations you can try - Bypass block resolution just to isolate whether credentials resolution is the issue: - Temporarily pass a plain token dict (only for a quick test; don’t keep secrets inline long-term):
Copy code
gitlab_repo = GitRepository(
      url="<https://gitlab.devops.prdroot.net/service/dwh/data-science-no/production-models/norway-pkt.git>",
      credentials={"access_token": "<PAT>"},
      branch="dev",
  )
If this works, the failure is in block resolution (PREFECT_API_* missing, wrong workspace, or the block doesn’t exist in that workspace). - If your GitLab has a private CA, add it and rebuild image as noted above. If you can share: - Output of the manual 
git ls-remote
test from inside the pod - Output of 
env | grep PREFECT_API_
- The DEBUG-log line that follows “Failed to clone repository …” (Prefect often appends a hint like “Authentication failed”, “Could not resolve host”, or “SSL certificate problem”) …I can pinpoint the exact fix. My current bet is either private CA trust or the token not having permissions to that repo/branch at runtime.
4 Views
Open in Slack
PreviousNext
Powered by