https://prefect.io logo
Join Slack
Powered by
Having a difficult time interpreting how this flow...
# prefect-ui
y
Having a difficult time interpreting how this flow definition turns into this graphic. Can anyone help? How would I make it show that task 2 depends on task 1, and log_task depends on task 1 & task 2, and task 4 depends on log_task? I'm really unclear on what this radar view is trying to tell me
o
Hey, You need to use .submit if you want to visualize a dependency graph.
plus one 1
j
Yeah I had the same issue and the solution was what Oscar proposed. But I think this should be fixed somehow if possible, would be surprised if this is the intended behaviour
o
Agreed!
t
Hi again Yusuf! These docs should give you a good example of how to use 
.submit()
syntax in your flow code. https://docs.prefect.io/tutorials/execution/?h=.submit#result
y
Thank you for the comments, all! This is how it looks now: The answer here in the discourse may need to be updated.
z
We do track dependencies without using 
submit
but it depends on the data that you’re passing
🙏 2
k
@YSF I recently ran into this same confusion and explored how it works in detail over here: https://prefect-community.slack.com/archives/C0192RWGJQH/p1661609573510969
f
Same here, just want to shared feedback : I’m having a very hard time trying to understand the radar, and how to show dependencies between tasks visually. I believe a UI is also some sort of documentation on what a flow does. I was desperately looking for documentation and tutorial about the radar, but I guess I finally have a clue with this thread and the submit() method. Weird though that the radar doesn’t show dependency natively Tx
This is my final workaround, submit alone doesn’t work. We also need to specify wait_for. It’s a lot of useless code just in order to have a radar that displays the tasks in order. @Zanie , I’m curious to see if you have a better way to track dependencies without using .submit
Copy code
@flow(task_runner=SequentialTaskRunner())
def rfdm():
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"I am an INFO message")
    t1 = ingest_data.submit()
    t2 = dbt_run.submit(wait_for=[t1])
    t3 = a.submit(wait_for=[t2])
    t4 = b.submit(wait_for=[t3])
t
To chime in here, I feel like whomever designed this radar was trying to be a little too edgy perhaps (hopefully that doesn’t come across too rude). I think this would be far better it it were redesigned to be a simpler “DAG-style” graph. Not mentioning any competitors, but this is the part of Prefect that I really find lacking. If you have simple flows it isn’t an issue. But if you have even a semi-complicated graph of dependencies, this visualization is critical IMO. Especially if you want to start eg parsing large dbt files and visualise the dependencies. Does anyone know if improvements to this are on the horizon? This is one of my main gripes to adopting Prefect.
➕ 5
z
cc @justabill
🙏 1
a
thanks for the feedback, Todd, we have a work in progress to add alternative visualizations meanwhile, for dbt, you could use dbt diagram and if you want to parse it into separate tasks, check out this blog post series https://medium.com/the-prefect-blog/how-to-build-a-modular-data-stack-data-platform-with-prefect-dbt-and-snowflake-89f928974e85
t
Thanks, @Anna Geller. Using the dbt diagram means that you have only a fragmented view of the process if there are upstream and downstream non-dbt tasks. So it doesn’t really provide the solution I am looking for or expecting. Any rough ETAs for these new visualisations or a beta program for them that I can assist with? More than happy to trial run them and provide feedback, identify bugs etc.
a
that's kind of you, I appreciate that. We'll keep you posted via #CKNSX5WG3 and we'll keep in mind that you're interested as early tester
b
@YSF Am I right in assuming you expect this flow to end up looking something like this?
y
Yeah exactly. The thing that's immediately clear here is that log_task has a dependency on both task 1 and task 2. In the way it renders in the radar chart as in my screen grab above after I used the correct syntax it still takes a lot of visual effort and its still not unmistakeably clear. If I didn't know better I'd have thought that task 1 was generating either two outputs or the same output and then was being joined to the output of task 2 before that single object was fed into log_task. Although full disclosure we just ended up signing a contract with Prefect 1 and will switch in the future.
b
Ahh, interesting. Nice, thank you.
o
If you run your tasks using submit and wait_for, it will build the graph the way you're expecting it to work. I'm not sure why submit is required, perhaps that's something that could be improved.
gratitude thank you 1
z
Running your tasks linearly does not mean there is necessarily a dependency between them
If you pass data between tasks, we’ll show a relationship
If you run your tasks concurrently (i.e. with submit) and want to have a task wait for another task, you can use 
wait_for
to indicate an implicit relationship.
o
Makes sense. Could it be made to work without submit, though? I mean, maybe wait_for could be all that's required to build the implicit relationship?
z
wait_for
doesn’t have the same meaning outside of 
submit
since things are already running sequentially, we can definitely investigate a way to indicate implicit relationships in sequential code though!
o
I think that would be pretty neat!
plus one 1
t
Is the timeline view per this release the proposed alternative visualisation to solve the above problem? Or are there additional visualisations in the pipeline as well? https://prefect-community.slack.com/archives/CKNSX5WG3/p1673555396413039
z
Additional visualizations are being planned as well
👌 1
🔥 1
f
wow that’s really cool !
3 Views
Open in Slack
PreviousNext
Powered by