Josh Greenhalgh
02/03/2021, 8:02 PMZanie
Zanie
import time
import random
import prefect
from prefect import Flow, task, Parameter
MAX_SIZE = 30
@task
def dynamic_list_of_tasks(window_size):
size = random.randint(window_size, MAX_SIZE)
size += size % window_size # Ensure it is divisable for the fixed window
return [i for i in range(size)]
@task
def fixed_window(tasks: list, window_size: int):
windowed = []
while tasks:
windowed.append(tasks[:window_size])
tasks = tasks[window_size:]
return windowed
@task
def display(i):
<http://prefect.context.logger.info|prefect.context.logger.info>(f"Processing {i}")
window_size = Parameter("window_size", default=2)
with Flow("windowed-tasks") as flow:
tasks = dynamic_list_of_tasks(window_size)
windowed_tasks = fixed_window(tasks, window_size)
display.map(windowed_tasks)
flow.run()
# flow.register("default")
Josh Greenhalgh
02/03/2021, 8:03 PMStartFlowRun
task for around 100 different param combinations but only want say 10 mak at a time to runJosh Greenhalgh
02/03/2021, 8:03 PMZanie
Josh Greenhalgh
02/03/2021, 8:03 PMZanie
Zanie
Zanie
Zanie
Josh Greenhalgh
02/03/2021, 8:05 PMJosh Greenhalgh
02/03/2021, 8:11 PMJosh Greenhalgh
02/03/2021, 8:12 PMZanie
Josh Greenhalgh
02/03/2021, 8:15 PMIf creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.
Josh Greenhalgh
02/03/2021, 8:15 PMZanie
Josh Greenhalgh
02/03/2021, 8:17 PMJosh Greenhalgh
02/03/2021, 8:17 PMJosh Greenhalgh
02/03/2021, 8:17 PM