Black Spy

    Black Spy

    2 months ago
    Hi, Can you help me to do create dynamic DAG flow based on configuration file(Yaml) and also how to do parallel scheduling at the sametime @Anna Geller
    Anna Geller

    Anna Geller

    2 months ago
    I think the best example I could give you is the prefect-openmetadata collection which does exactly what you described - it generates a flow based on YAML config https://prefecthq.github.io/prefect-openmetadata/ Here is an example - the
    ingest_metadata
    is a function decorated with flow:
    from prefect_openmetadata.flows import ingest_metadata
    
    config = """
    source:
      type: sample-data
      serviceName: sample_data
      serviceConnection:
        config:
          type: SampleData
          sampleDataFolder: "example-data"
      sourceConfig: {}
    sink:
      type: metadata-rest
      config: {}
    workflowConfig:
      openMetadataServerConfig:
        hostPort: <http://localhost:8585/api>
        authProvider: no-auth
    """
    
    if __name__ == "__main__":
        ingest_metadata(config)
    there are many more examples in the docs here