https://prefect.io logo
#prefect-community
Title
# prefect-community
b

Black Spy

07/04/2022, 7:37 AM
Hi, Can you help me to do create dynamic DAG flow based on configuration file(Yaml) and also how to do parallel scheduling at the sametime @Anna Geller
1
a

Anna Geller

07/04/2022, 11:51 AM
I think the best example I could give you is the prefect-openmetadata collection which does exactly what you described - it generates a flow based on YAML config https://prefecthq.github.io/prefect-openmetadata/ Here is an example - the
ingest_metadata
is a function decorated with flow:
Copy code
from prefect_openmetadata.flows import ingest_metadata

config = """
source:
  type: sample-data
  serviceName: sample_data
  serviceConnection:
    config:
      type: SampleData
      sampleDataFolder: "example-data"
  sourceConfig: {}
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  openMetadataServerConfig:
    hostPort: <http://localhost:8585/api>
    authProvider: no-auth
"""

if __name__ == "__main__":
    ingest_metadata(config)
there are many more examples in the docs here
5 Views