Amogh Kulkarni

    Amogh Kulkarni

    8 months ago
    Hi All, I have been using Prefect for over 6 months and it’s working great. I am using the run_config as Kubernetes and storage as S3. Currently, the developers check-in the code into github.  The structure of my github repo is something like this: repo_name ----flows --------flow1.py --------flow2.py ----lib --------etl.py --------reverse-etl.py dockerfile If the developer, checks-in the dag in flows folder, the CI registers the flow to prefect cloud and if the developer checks in a python file in lib folder, the CI builds the docker image and pushes it to AWS ECR. The lib folder is a python module which has the resuable code needed to run the dags. I am now trying to create a seperate micro-service which will only register flows.  I realized that there is no way that Prefect allows you to only register a flow without having access to the python file which has the flow and it going over the entire code of the flow file. Is there a way I can register the flow from a remote server?
    Kevin Kho

    Kevin Kho

    8 months ago
    I think your blocker is that the custom module in the lib folder is not available during registration time? Or is the flow file itself not available as well? Because if it’s just the lib folder, you can work around this my importing that module inside tasks so that importing is deferred to runtime. If you don’t even have the flow file itself, it becomes very hard. First, the flow is serialized into a string representation, and then registered with an API call. If you don’t have the Flow, you at least need the string representation, which is quite hard to work with. But without the file, I’m not sure where you would get this string representation from other than constructing it somewhere else, which seems a bit brittle of a design. This discussion may shed some insight if you haven’t seen it yet
    But I am not sure you can make it as tiny of a micro-service as your are envisioning (really not sure, not trying to discourage)
    Amogh Kulkarni

    Amogh Kulkarni

    8 months ago
    Thanks @Kevin Kho I will go through the discussions and I will get back to you if I have any additional questions.
    Since we are checking in the flow file in github, we have access to the Flow. importing the lib module within tasks makes a lot of sense. I was thinking of creating a private python package and installing that in the container of the registration microservice. But that involves lot of config and version management. Before doing tht i thought of checking with Prefect if there is an easy straightforward way to handle this. Thanks a lot for your input and sharing the discussion thread. I will have to take a step back and rethink the overall design.
    Kevin Kho

    Kevin Kho

    8 months ago
    Ah ok I see. Yes the import in a task might be the easiest way.