Thread
#prefect-ui
    Dahn Jahn

    Dahn Jahn

    1 year ago
    Hi all! I am attempting to use Prefect to standardize and simplify my team's ETL processes. One of the main problems is that it's never clear which script uses which files and which files are its outputs. I thought Prefect could help solve that with the DAG visualization, but I did not manage to get that to work: only input parameter names are visualized, and I don't see a way to visualize an output file. One hacky solution I came up with would be to have all paths (inputs and outputs) as Parameters with default values equal to the actual filepath. If I could then visualize the default parameter value in the DAG, I would have a working solution. However, I am not sure if I can visualize the default parameter values. Perhaps I can inherit from Task/Parameter and override its name to reflect that information? How could I do it, or is there a different solution altogether? Ideally, the graph would look something like
    input_filepath
    ->
    function
    ->
    output_filepath
    .
    Jenny

    Jenny

    1 year ago
    Hi @Dahn Jahn we have a PR open to show task run names in the schematic in the UI. Once this is released I think you could update the task run names to show your values. Something like down here in the docs: https://docs.prefect.io/core/idioms/task-run-names.html#naming-task-runs-based-on-inputs
    Dahn Jahn

    Dahn Jahn

    1 year ago
    @Jenny thank you! Can you point me to the PR, please? I was half hoping that I'm misunderstanding Prefect and there's a native way to represent files in the DAG, but if that's not the case, then overriding the parameter names is the second best thing.
    Jenny

    Jenny

    1 year ago