https://prefect.io logo
Title
n

Nick Williams

02/13/2023, 1:50 PM
Hi all, really enjoying Prefect so far. Much easier than airflow and much better suited to my requirements at work. I have a number of pipelines which load from Postgres into BigQuery, performing standard transformations along the way. What I've done is create a flow which loads a given JSON file from a bucket and then loads that table with the settings given in the JSON file. This flow can be run on its own with a parameter specifying the JSON filename. I have another flow which loops through the bucket and launches the single-table subflow for each file in the bucket, so I can schedule all the tables I need on a daily basis. It all works fine and I'm really happy with this. However, each subflow has a generic "active-pig", "tested-puma" etc name. Is there any way that I can set the names of these subflows in the python task decorator? It would give me improved visibility of which tables have been run.
1
p

Peyton Runyan

02/13/2023, 2:04 PM
If you go down a bit you'll see how to do dynamic names
👍 1
n

Nick Williams

02/13/2023, 2:05 PM
Brilliant! Thank you