Anyone here from the prefect-databricks contributo...
# prefect-contributors-archivede
Edmondo Porcu
10/19/2023, 9:49 PMAnyone here from the prefect-databricks contributor? I do have an important question 🙂
n
Nate
10/19/2023, 10:33 PMhey @Edmondo Porcu - I've done a lot of work on the collections (not that one exactly), i can probably help. What's up?
e
Edmondo Porcu
10/19/2023, 10:46 PMthere is a flow that we use, “job runs submit and wait”. It uses:
1. one task to submitting the job on databricks
2. one flow that will poll the job until it’s done creating tasks for polling
Some of our jobs are really long running, and we don’t have any way to make the job id that is created by submitting the job available on prefect UI. One obvious solution would be to add a parameter to the current job and introduce an “interceptor”, however this would break the signature because it uses a kwargs argument
Edmondo Porcu
10/19/2023, 10:47 PMfor example we were considering using artifacts and log this value as an artifact.
https://github.com/PrefectHQ/prefect-databricks/blob/main/prefect_databricks/flows.py#L251
I was considering making the change myself an opening a PR to prefect-databricks, but I don’t really know how to add an interceptor without breaking the signature
n
Nate
10/19/2023, 11:01 PMOne obvious solution would be to add a parameter to the current job and introduce an “interceptor”, however this would break the signature because it uses a kwargs argumentit might just be me being naive about databricks but can you clarify what you mean by this? my gut reaction is that we could add a
handler: Optional[Callable]e
Edmondo Porcu
10/20/2023, 6:18 PMit’s exactly right
Edmondo Porcu
10/20/2023, 6:19 PMOur users on Prefect expect the job id on Databricks/the url to be available for troublehsooting and visiting the Databricks url. If a job lasts 10 hours, it becomes available after 10 hours.
I would love the handler parameter. won’t that however be a breaking change to the python signature?
n
Nate
10/20/2023, 6:50 PMI don't believe an optional kwarg added to the flow would necessarily be a breaking change, since if you aren't aware of the change and do not provide a
handlerenhancemente
Edmondo Porcu
10/27/2023, 12:53 AMEdmondo Porcu
10/27/2023, 8:21 PMThanks @Nate how does the release cycle for the tasks / flows and work?
n
Nate
10/27/2023, 8:23 PMwe don't have a cycle for the collections at this point, its just basically when something new gets added that people want we'll cut a release, like in this case
so I'll put a PR up to update release notes (or you could do that quick before we merge this, no biggie), get that merged and then I can cut a release
e
Edmondo Porcu
10/27/2023, 8:26 PMđź’Ż
Edmondo Porcu
10/27/2023, 8:26 PMthanks
n
Nate
10/27/2023, 9:36 PMhey @Edmondo Porcu - v0.2.1 is released!
e
Edmondo Porcu
11/09/2023, 8:49 PM@Nate one of the problem we face with subflows is that when we log artifacts, these are logged in the Databricks subflow (wait something) rather than in the parent flow
Edmondo Porcu
11/09/2023, 8:49 PMCan we retrieve the parent flow context in Prefect?
n
Nate
11/09/2023, 8:50 PMgenerally we suggest you pass what you need from parent contexts to sub contexts
e
Edmondo Porcu
11/09/2023, 8:52 PMyeah that makes sense, we are going to rertrieve the current flow id
n
Nate
11/09/2023, 8:55 PMđź‘Ť