I wanted to thank @Nate for a convo today. He solved an issue I have been trying to fix for a couple of days.
I opened an issue at @Nate proposition. Hopefully the small but useful functionality I was after gets implemented within prefect in future versions