Hi all, really enjoying Prefect so far. Much easie...
# prefect-getting-started
n
Hi all, really enjoying Prefect so far. Much easier than airflow and much better suited to my requirements at work. I have a number of pipelines which load from Postgres into BigQuery, performing standard transformations along the way. What I've done is create a flow which loads a given JSON file from a bucket and then loads that table with the settings given in the JSON file. This flow can be run on its own with a parameter specifying the JSON filename. I have another flow which loops through the bucket and launches the single-table subflow for each file in the bucket, so I can schedule all the tables I need on a daily basis. It all works fine and I'm really happy with this. However, each subflow has a generic "active-pig", "tested-puma" etc name. Is there any way that I can set the names of these subflows in the python task decorator? It would give me improved visibility of which tables have been run.
1
p
If you go down a bit you'll see how to do dynamic names
👍 1
n
Brilliant! Thank you