Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey guys, checked my overnight flows and they failed with an error on the databricks side; however, when I restart then they fail right away with no info in the logs on the cloud. Is there any known issues at the moment? All my agents are active - run id = 5fbc05b3-d2e3-43ee-8d5d-842aaa04e23b

Further info - I started another run on another flow via the same agent and all is fine so far - this run id is:

f3233bf6-1cdb-4260-9a03-0c3259bc0b3a

the first run ID doesn't exist on our side - is it a typo?

and thanks for the follow-up - that's also something I wanted to suggest - whether you could start a new flow run instead of restarting the current one

this is defo the run id - i missed the 5 sorry :stuck_out_tongue::

<https://cloud.prefect.io/aggreko-data/flow-run/5fbc05b3-d2e3-43ee-8d5d-842aaa04e23b>

Here is another two we are having the same issue on:

<https://cloud.prefect.io/aggreko-data/flow-run/107a3153-ef3e-4657-a969-b6aa6396f41e>

<https://cloud.prefect.io/aggreko-data/flow-run/324b7fc4-24e2-411e-9d39-65ead86f627a>

We are just unable to restart them.

did you configure results? this is required for restarts to work

No this is not something I have done - however, my previous attempts at restarts usually work and I have not made any changes to my configuration in a long time....

Screenshot 2022-05-25 at 13.10.53.png

This what I am seeing in the logs when trying to restart:

I see no logs here other than state updates in the first one - hard to understand what is the root cause here

the second run only shows that some Spark jobs failed

exactly, I see the same - it looks like some Spark jobs are failing? did you check why this might be happening?

Nothing is ever sent to databricks after I hit restart...

restart functionality is to restart a flow run from a failed task run - given that your work is not really happening on the Prefect side, but rather on Spark, it might be more helpful to start a new flow run rather than restart to restart your data processing process

processing process :exploding_head: :smile:

Okay cool - just a bit strange as it used to allow us to restart from the failed task but we have lost that functionality now :disappointed:

Really strange as this one restarted from some failed spark jobs:

<https://cloud.prefect.io/aggreko-data/flow-run/bd4a0de0-b1d7-411d-a361-090af4e047f5?overview>

Checking after holidays: LMK if this is still an open issue

Nah this was more of an observation on Prefect allowing us to restart even tho we had not changed any config when it never let us on the previous run! Thanks for coming back to me :slightly_smiling_face:

Awesome, thanks for the update, appreciate it!