04/18/2023, 7:45 AM
Hi, I'm creating a product (a kind of uptime kuma alternative) which will ship prefect for scheduling, an API (FastAPI), a frontend (Nuxt3), a database (PSQL >= 15, with 2 db: prefect + my db), and N prefect agents. The issue I'm facing is I want to share flow code from the shipped product (not from an external github repo or aws bucket as example), so I'm wondering if the solution couldn't be serve the code from my API ? That would mean I have to create an HTTP storage block. • Is it the right solution ? • if yes, any tips about creating and registering an HTTP storage block ? • should I open an issue to debate about if this feature is suitable for prefect itself or it's too specific for my usecase ?

Ryan Peden

04/18/2023, 4:33 PM
My thoughts/replies: • It's "right" in that you can make it work. It might be cleaner/easier to use a self-hosted S3-compatible object store like MinIO, but its AGPL license might make it unsuitable for use in your project. • Since you're using agents, I assume you're working with deployments. So at a minimum, if your agents only need to pull code from storage, your block would need to implement the
interface. See the GitHub block for an example of a storage block that does this. • You could probably implement that interface with a block that stores an HTTP endpoint as one of its fields, and then calls the endpoint, pulls down a zip containing the directory, and then unzips it in the correct location. • If your block needs to push code to storage as well it will need to implement the
interface, which adds a
method. • If your product is in its own Python module that gets installed via pip, you can get Prefect to auto-load it by setting the right entrypoint. Prefect will load anything with an entrypoint of
, and will also load any additional entrypoints specified via the
config setting. (This section of code is where the autoregistration happens). • See here for an example of a module that sets an entrypoint. I think this is probably too specific to your use case to be part of Prefect itself. You'd need a member of the Prefect team to verify, however. But if you think others would like/want to use it, you could easily publish it as a PyPI module. There's a Prefect collection template generator that can help you get started on that quickly.