@Marvin I'm getting the following error in the deployment when I set

include_submodule: true

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 309, in coroutine_wrapper
    return call()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 123, in git_clone
    await storage.pull_code()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 233, in pull_code
    await self._clone_repo()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 261, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://github.com/spanio/data_science.git>' with exit code 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 426, in retrieve_flow_then_begin_flow_run
    else await load_flow_from_flow_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 307, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone

Your message was too long by 490 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

I'm getting the following error in the deployment when I set

include_submodule: true

```Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps step_output = await run_step(step, upstream_outputs) File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step result = await from_async.call_soon_in_new_thread( File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult return await asyncio.wrap_future(self.future) File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 309, in coroutine_wrapper return call() File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in call return self.result() File "/usr/local/lib/python3.10/site-packages/prefect

@Marvin I'm getting the following error in the deployment when I set

include_submodule: true

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 426, in retrieve_flow_then_begin_flow_run
    else await load_flow_from_flow_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 307, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone

It seems like there isn't a direct solution available from the Prefect documentation or GitHub issues. Given that the error occurs while running the

git_clone

step with

include_submodule: true

and the traceback points to an issue with pulling from the remote, you can try the following steps to troubleshoot and potentially resolve the issue: 1. Check Remote Repository URL: Ensure the repository URL and references (e.g., branches, tags) are correct in your deployment configuration. 2. Git Submodule Initialization: Make sure the submodule is correctly initialized and updated in the remote repository. You can manually test this by cloning the repository and running:

sh
    git clone --recurse-submodules <repository_url>

3. Update Deployment Configuration: Verify that the deployment YAML file configuration aligns with the requirements for including submodules. Sometimes, specifying the correct paths and ensuring all dependencies are properly handled can make a difference. 4. Check Permissions: Ensure that the authentication method you're using has the necessary permissions to access submodules. 5. Verbose Logging: Enable verbose logging to get more insights into what might be failing during the git clone step. For instance, you can modify the deployment script to capture and print detailed error messages. For further in-depth assistance, you might want to open an issue on the Prefect GitHub repo with detailed information on your deployment configuration and the error.

@Marvin where's the source code for include_submodule: true

The source code related to

include_submodule: true

for integrating GitHub repositories with Prefect can be found in the

prefect_github/repository.py

module. It handles querying and managing submodules within a repository. Here's a relevant excerpt from the Prefect documentation:

# Document metadata
title=None link='<https://docs.prefect.io/latest/integrations/prefect-github/repository/>' source='url' document_type='web page'
...
Source code in prefect_github/repository.py
- ...
- query_repository_submodules
- async
Returns a list of all submodules in this repository parsed from the .gitmodules file as of the default branch's HEAD commit.
- Parameters:
  - owner | str

You can access the complete source code and understand how

include_submodule

is implemented by visiting the following link: Prefect GitHub Integration Documentation

@Marvin the error

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 309, in coroutine_wrapper
    return call()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 123, in git_clone
    await storage.pull_code()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 233, in pull_code
    await self._clone_repo()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 261, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://github.com/spanio/data_science.git>' with exit code 1.

Your message was too long by 255 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

the error

```Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps step_output = await run_step(step, upstream_outputs) File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step result = await from_async.call_soon_in_new_thread( File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult return await asyncio.wrap_future(self.future) File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 309, in coroutine_wrapper return call() File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in call return self.result() File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result

the error:

Flow could not be retrieved from deployment.
    return call()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 123, in git_clone
    await storage.pull_code()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 233, in pull_code
    await self._clone_repo()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 261, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://github.com/spanio/data_science.git>' with exit code 1.

Your message was too long by 39 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

the error:

```Flow could not be retrieved from deployment. return call() File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in call return self.result() File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result return self.future.result(timeout=timeout) File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async result = await coro File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 123, in git_clone await storage.pull_code() File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 233, in pull_code await self._clone_repo() File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line

@Marvin the error:

Flow could not be retrieved from deployment.

  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 123, in git_clone
    await storage.pull_code()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 233, in pull_code
    await self._clone_repo()
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 261, in _clone_repo
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://github.com/spanio/data_science.git>' with exit code 1.

Your error traceback reveals that the issue lies within the

git_clone

function in

prefect/deployments/steps/pull.py

, specifically when attempting to pull code from the repository. The error message "Failed to clone repository ... with exit code 1" indicates a failure when running the

git clone

command. This can be due to several issues: 1. Invalid Repository URL: Double-check that the repository URL is correct and accessible. 2. Access Permissions: Ensure that the authentication token or SSH key being used has the necessary permissions to access the repository. 3. Submodule Configuration: Ensure that your

.gitmodules

file and the submodules' URLs are correct and accessible. ### Steps to Troubleshoot: 1. Manual Clone: Manually clone the repository with submodules on your local machine to verify that the repository URL and submodule configuration are correct:

sh
   git clone --recurse-submodules <https://github.com/spanio/data_science.git>

2. Check Access: Ensure that the authentication method (token or SSH key) used in the Prefect deployment has the correct access permissions for both the main repository and its submodules. 3. Verbose Output: Enable verbose output in the Prefect deployment script to get more detailed error messages:

python
   import logging
   logging.basicConfig(level=logging.DEBUG)

4. Correct Submodule Reference: Verify that the submodule references in your

.gitmodules

file are correct and accessible. 5. Configuration in Prefect Deployment: Ensure that the

include_submodule: true

configuration in your deployment YAML is set correctly. Here's a sample snippet:

yaml
   storage:
     module: "prefect_github"
     include_submodule: true
     ...

If the problem persists, it might help to check Prefect's issue tracker for similar issues or raise a new issue with detailed information.