Sven Teresniak
09/11/2020, 8:49 AMtag
s for tasks, label
s for flows, but not possible to use tags or labels for flow-runs, right?
Now, the idea is to write a ressourcemanager as kind of Lock (reentrant or not, semaphore or whatever flavor). Then you could (hopefully) enter a RessourceManager with
-block the same way you would use a Lock
in Python code.
Something like
1. setup()
searches (like in "asking Prefect") for all flows-runs with a given tag (that is, a run with a flow that has this certain tag set)
2. if setup()
does NOT find another flow-run with that tag, the tag is set (i don't know yet where and how)
3. if setup()
DOES find find another matching flow-run (or RM): then do some polling or return with successful error (configuration)
4. then all the tasks in the RM's with
environment run (this is the part that should be synchronized/locked)
5. on exit the RMs cleanup()
is called to remove the tag/label/whatever from the current flow (to allow other flows to set the tag)
Do you think this is a good idea? Which/where is the best way to store that tag? Using the Context, Parameters, Tags, Labels? If you think this could be a nice feature we could create a prototype and pull request.
We need to prevent a race condition here and for that we need to make the execution of several task isolated (the I in ACID in RDBS)Matt Wong-Kemp
09/11/2020, 9:50 AMidempotency_key (str, optional): an idempotency key; if provided, this run will be cached for 24 hours. Any subsequent attempts to create a run with the same idempotency key will return the ID of the originally created run (no new run will be created after the first). An error will be raised if parameters or context are provided and don't match the original. Each subsequent request will reset the TTL for 24 hours.
(from https://docs.prefect.io/api/latest/client/client.html#client-3)Sven Teresniak
09/11/2020, 11:41 AMnicholas
09/11/2020, 1:58 PMSven Teresniak
09/11/2020, 2:22 PMnicholas
09/11/2020, 2:26 PMSven Teresniak
09/11/2020, 2:30 PMsetup()
method block until the current flow-run cannot find any other flow-run in the system that matches a certain tag (which is only available during runtime and based on scheduled time).nicholas
09/11/2020, 2:40 PMquery {
flow_run(where: { context: { _eq: # some json comparison } }) {
id
# other fields
}
}
You can use the InteractiveAPI to help put that query together.Sven Teresniak
09/11/2020, 2:43 PMnicholas
09/11/2020, 2:43 PMSven Teresniak
09/11/2020, 2:43 PMnicholas
09/11/2020, 2:43 PMSven Teresniak
09/11/2020, 2:45 PMlabel
fix from the time you register a flow, right?nicholas
09/11/2020, 2:46 PMSven Teresniak
09/11/2020, 2:46 PMnicholas
09/11/2020, 2:48 PMSven Teresniak
09/11/2020, 2:49 PM