Hello folks, I keep getting this error when trying to run a deployment in Prefect Cloud :

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 311, in coroutine_wrapper
    return call()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 123, in git_clone
    await storage.pull_code()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 233, in pull_code
    await self._clone_repo()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 241, in _clone_repo
    repository_url = self._repository_url_with_credentials
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 167, in _repository_url_with_credentials
    for k, v in credentials.items():
AttributeError: 'str' object has no attribute 'items'

I don't really know where the problem is coming from, as it seems to come from Prefect's modules and not something I wrote. Here is the .yaml of my deployment :

name: tenacy-prefect
prefect-version: 2.20.2

# build section allows you to manage and build docker images
build:

# push section allows you to manage if and how this project is uploaded to remote locations
push:

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: <https://my-private-gitlab-instance.io/data/tenacy-prefect.git>
    credentials: '{{prefect.blocks.secret.gitlab-personal-token}}'


# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: Younium
  version:
  tags: []
  description:
  entrypoint: flows/flows.py:younium_flow
  parameters: {}
  work_pool:
    name: tenacy-work-pool
    work_queue_name:
    job_variables: {}
  schedules: []

Hey! Could you try replacing

credentials: '{{ ... }}'

with

access_token: '{{ ... }}'

?

I tried, but I get this error instead :

RuntimeError: Failed to clone repository '<https://my-private-gitlab-instance.io/data/tenacy-prefect.git>' with exit code 128.

It seems that with

credentials

it managed to clone the repo but got stuck somewhere else.

With

credentials

it error'ed before it was able to even clone, because you're passing a

Secret

block instead of a

GitLabCredentials

block

I have the secret block and my token should be valid.

hey! sorry for delayed reply here. I'm glad you were able to get it working!

No worries, I imagine we're not in the same timezone. One last question, I'm having trouble picturing the purpose of Prefect Cloud if I still have to host my docker image somewhere. Using Prefect Cloud simply allows me not to have to host a database instance and the API, right ? I'm running the docker image like this :

docker run -e PREFECT_API_URL=$PREFECT_API_URL -e PREFECT_API_KEY=$PREFECT_API_KEY prefect-tenacy

It works on my machine, but I imagine it will work as well when we host it on our infra, because in that case it will be us calling the Prefect API, and not Prefect trying to clone our repo. Have I understood correctly ?

Correct, you can think of Prefect Cloud as a hosted version of the prefect orchestration server/db. The code execution however happens in your environment (which is when code is pulled, if you're using git remote storage or just loaded if you're using an image).

Okay, so the API hosted by Prefect Cloud will trigger the docker image hosted on our servers ? Which means we still need to whitelist Prefect's IP range ? edit : sorry for all those questions, I'm not used to handling the dataops side of data engineering 😛

No worries! Prefect uses a hybrid approach whether you are using prefect cloud or hosting the api yourself. Workers: • are long running process that lives in your infrastructure • it polls the api for new runs in your work pool • when it finds a new run it will deploy and execute the run to your infrastructure (defined by your work pool) Prefect isn't calling into your infrastructure directly, the worker and execution environments will be reaching out to prefect cloud. What you need to do for that depends on your network setup. We publish our ip here. It depends on your security setup but most commonly I think people whitelist the prefect api uri for outbound trarffic.

Thanks a lot for the link and the explanation !

np!