Hi all - has anyone used a Github storage block with a repo that has a git submodule? Is there a con...

Trent Shapiro

06/22/2023, 4:58 PM

Hi all - has anyone used a Github storage block with a repo that has a git submodule? Is there a convenient way to pull the git submodule at runtime?

✅ 1

Trent Shapiro

06/22/2023, 5:52 PM

Copy code

from prefect.filesystems import GitHub

submodule_block = GitHub.load("submodule-block-name")
submodule_block.get_directory(local_path=submodule_folder_path)

is what i'm doing now, but requires setting up a separate github block for the submodule. wish there was an option to add

--recurse-submodules --remote-submodules

to the initial repo clone

Bianca Hoch

06/22/2023, 7:51 PM

Hi Trent! I'm not sure about the Github storage block having that functionality, but it looks like you should be able to include git submodules when cloning a repository via the prefect.yaml pull action.

Bianca Hoch

06/22/2023, 7:56 PM

It'd look something like this example:

Copy code

pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/org/repo.git>
        include_submodules: true

Trent Shapiro

06/22/2023, 7:59 PM

thanks for this! I'll definitely look into it. Right now I'm managing the deployments through a separate script, but it looks like

prefect.yaml

is way more condensed. much appreciated

Trent Shapiro

06/22/2023, 8:01 PM

is this relatively new? on 2.10.12 and dont see

prefect init

as an option

Bianca Hoch

06/22/2023, 8:04 PM

ah yes, i think you need a newer version

👍 1

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.