Hi all! I am attempting to use Prefect to standardize and simplify my team's ETL processes. One of the main problems is that it's never clear which script uses which files and which files are its outputs. I thought Prefect could help solve that with the DAG visualization, but I did not manage to get that to work: only input parameter names are visualized, and I don't see a way to visualize an output file.
One hacky solution I came up with would be to have all paths (inputs and outputs) as Parameters with default values equal to the actual filepath. If I could then visualize the default parameter value in the DAG, I would have a working solution. However, I am not sure if I can visualize the default parameter values. Perhaps I can inherit from Task/Parameter and override its name to reflect that information? How could I do it, or is there a different solution altogether? Ideally, the graph would look something like
@Jenny thank you! Can you point me to the PR, please? I was half hoping that I'm misunderstanding Prefect and there's a native way to represent files in the DAG, but if that's not the case, then overriding the parameter names is the second best thing.