Hi Everyone, We are using Prefect's imperative API , and wanted to use the GitHub storage option. A...

Omar Sultan

08/02/2021, 8:23 AM

Hi Everyone, We are using Prefect's imperative API , and wanted to use the GitHub storage option. All the tutorials refer to the use of the functional APIs when registering and storing script based Flows. If anyone can point me in the right direction this will be really appreciated

Yueh-Han

08/02/2021, 10:02 AM

Hello Omar - good question, and thanks for pointing out the tutorial is written for functional API.

Yueh-Han

08/02/2021, 10:05 AM

So to add details to your question, ~~is the bottleneck on loading flow dependencies from storage? Assuming that you’re able to load the task from GitHub storage, but the logic is still defined in your execution files~~ see kevin’s comment below

Kevin Kho

08/02/2021, 1:49 PM

Hey @Omar Sultan, it would just be the

flow.storage = GitHub(…)

also with the imperative API.

Yueh-Han

08/02/2021, 2:11 PM

I think @Kevin Kho get it right here. Once your flow is defined in your file - you can “imported” it with Prefect Storage - doesn’t matter if it’s defined with functional API or imperative API, same flow object in the end.

Omar Sultan

08/04/2021, 2:10 PM

Thanks guys for the update, I gave it a try with the option @Kevin Kho proposed. I was getting errors when trying to run the flow from another agent. We were short on time so didn't get a chance to debug this further. We went with the webhook option and are now storing them in data platform's shared storage so our API and its working perfectly

Omar Sultan

08/04/2021, 2:11 PM

will try again the github option at a later stage in the project but for now , i think we will stick with the Webhook option

Kevin Kho

08/04/2021, 3:07 PM

Are you doing something like this ? The authentication should be a string of the Secret name.

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.