Channels
Hello! I have a v1 flow which ingests data via SFT...
# ask-communityb
Ben Ayers-Glassey
03/30/2023, 3:56 PMHello! I have a v1 flow which ingests data via SFTP.
I have a task which lists the files to be downloaded, and then a mapped task which downloads each file.
The flow completed successfully, but seems to have not downloaded a few files.
There were 3,202 files, but only 3,197 seem to have been downloaded. So we're missing 5.
There are no errors associated with this; rather, it appears that only 3,197 mapped runs were ever scheduled, even though it says "Expected Runs: 3,202".
When I mouse over the "3,202", a popup says:
The number of mapped children expected to run.
Note that the number of active mapped runs may be less than this if some have not yet entered aThis seems like it would make sense while the flow was still running, but not once it has completed. 🤔state.Pending
👀 1
Ben Ayers-Glassey
03/30/2023, 3:57 PMThe logs for the mapped task itself don't show any issues:
Ben Ayers-Glassey
03/30/2023, 3:58 PMIf I click on its "Mapped Runs", I see:
1-25 of 3198...i.e. the 3197 mapped runs, plus the parent. So I can't find any mention of the 5 missing expected runs.
b
Bianca Hoch
03/30/2023, 4:54 PMHey Ben! If possible, I'd try running the following query to see if you can get the IDs of those task runs:
Copy code
query {
task_run(
where: {
flow_run_id: {_eq: "insert_flow_run_id"},
state: {_ne: "Success"}
}
) {
name
id
state
}
}Bianca Hoch
03/30/2023, 4:55 PM^ or something similar to this query. Essentially, it's looking for any task runs from that flow run which do not have a status of
SuccessBianca Hoch
03/30/2023, 4:59 PMOther than that, it's hard to say for certain what happened here if the logs don't tell the story of the missing work
b
Ben Ayers-Glassey
03/30/2023, 5:03 PM Copy code
In [11]: from prefect.client import Client
...:
...: c = Client()
...:
...: c.graphql("""query {
...: task_run(
...: where: {
...: flow_run_id: {_eq: "45ddf3e0-8d23-460e-a244-28fcb7b96f52"},
...: state: {_neq: "Success"}
...: }
...: ) {
...: name
...: id
...: state
...: }
...: }""")
...:
Out[11]:
{
"data": {
"task_run": [
{
"name": null,
"id": "ebad8ee0-51af-49e3-a519-a53e8b4b1366",
"state": "Mapped"
}
]
}
}Ben Ayers-Glassey
03/30/2023, 5:04 PMI think this makes sense... that's presumably the mapped task itself.
Its status says "Mapped" on the UI as well:
Ben Ayers-Glassey
03/30/2023, 5:05 PM...but the query doesn't seem to have found any of its children, which presumably means they all have "Success" status. Which again matches what I see in the UI.
Ben Ayers-Glassey
03/30/2023, 5:05 PMSo then, there is no clue why the number of children ended up less than "Expected Runs"?
b
Bianca Hoch
03/30/2023, 5:56 PMHmm..maybe we need to try re-configuring the query to get more details here
Bianca Hoch
03/30/2023, 5:57 PMIf you go to the interactive API, there is a "Query.mapped_children" option. It takes a task run ID as a search parameter
👀 1
b
Ben Ayers-Glassey
03/30/2023, 5:57 PMThis mapped task is supposed to store the files it downloads in GCS, and I'm fairly sure I see them all there -- all 3,202 files.
So it seems like maybe all of the task's mapped runs did actually run, just Prefect didn't learn about all of them?..
👀 1
🤔 1
b
Bianca Hoch
03/30/2023, 5:58 PMOdd, but it's nice that the files are there after all
b
Ben Ayers-Glassey
03/30/2023, 5:58 PMYes 🙂
Ben Ayers-Glassey
03/30/2023, 6:01 PM Copy code
query {
mapped_children(task_run_id:"45ddf3e0-8d23-460e-a244-28fcb7b96f52") {
min_start_time
max_end_time
state_counts
}
} Copy code
{
"data": {
"mapped_children": {
"min_start_time": null,
"max_end_time": null,
"state_counts": {}
}
}
}3 Views