Slackbot
06/13/2022, 6:47 PMDaniel Lomartra
06/16/2022, 4:47 PM@task(name="Skipped Task Branch")
def skipped_task_branch(skip_block_name):
logger = context.get("logger")
<http://logger.info|logger.info>(f"Sucessfully skipped {skip_block_name}")
raise signals.SUCCESS
Flow:
with Flow(
name="example flow"
) as flow:
myCondition = Parameter(name = "Run Tasks?", default = True)
with case(myCondition, True):
task1 = do_some_stuff()
task2 = do_some_other_stuff()
with case(myCondition, False):
skip = skipped_task_branch("task block name")
example_merge = merge(task2, skip)
Isaac Kargar
06/17/2022, 7:16 AMSurawut Jirasaktavee
06/19/2022, 10:47 AMsearch_run.py
and imported it to prefect_deploy.py
and stuck with the error below.Tomas Borrella
06/20/2022, 12:29 PMupstream_tasks
) and at the end of all of them, I would like to execute a last flow only if it is a specific day of the week (for example execute it the last of the flows only on Mondays). I have tried the following code, but the condition is always false
.... what is the good way to do this?
with Flow('prefect_parent', run_config=config.RUN_CONFIG) as prefect_p_flow:
a = flow_a()
b = flow_b(upstream_tasks=[a])
c = flow_c(upstream_tasks=[b])
if date.today().weekday() == 0: # Monday
d = flow_d(upstream_tasks=[c])
prefect_p_flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=4)
NOTE: I have been looking at Schedules
, but I don’t want the task to run at a specific point in time (because previous flows may take more or less), I would like it to run at the end of the previous task, but only if the day condition is met.
Any suggestions are welcomeEdmondo Porcu
06/21/2022, 12:00 AMFlorian Guily
06/21/2022, 8:29 AMIbrahim Sherif
06/23/2022, 11:30 PMSylvain Hazard
06/24/2022, 9:07 AMTravis Leleu
06/27/2022, 2:47 AMRohit
06/27/2022, 11:38 AMmartin hablak
06/29/2022, 8:38 AMflow_run_reference = flow.run()
for result in flow_run_reference.task_results:
yield result.json()
basically stream tasks result using say fastapi as soon as it finishes not waiting until whole flow finishTom Matthews
07/01/2022, 9:42 AM@flow
def pipeline(request):
# bunch of tasks, some of which run in parallel
# flow and tasks defined in another script
return output
...
def some_grpc_api_endpoint(request):
return pipeline(request)
Sergey Goncharov
07/01/2022, 3:32 PMJordan Charlier
07/01/2022, 3:54 PMChu
07/05/2022, 6:41 PMKevin Focke
07/06/2022, 3:37 PMKevin Focke
07/06/2022, 3:48 PMredsquare
07/06/2022, 4:00 PMValeria Romero
07/07/2022, 2:53 PMtask_definition_arn
when using Docker
storage". We use a task definition arn because the task definition is created with Terraform. Maybe we're going about this the wrong way?
Thanks in advance!Stefan
07/10/2022, 10:42 AMFlorian Guily
07/11/2022, 12:56 PM"dev", "staging", "prod"
?Luis Echegaray
07/12/2022, 4:01 PMAn error occurred (ThrottlingException) when calling the RegisterTaskDefinition operation (reached max retries: 2): Rate exceeded
Is there a prefect construct that will help us queue up the work or do people usually handle this programatically?Jon Ruhnke
07/12/2022, 4:22 PMMichiel Verburg
07/14/2022, 2:36 PMJohn
07/18/2022, 2:25 PMJess Zhang
07/19/2022, 1:01 AMJess Zhang
07/19/2022, 1:01 AMChu
07/19/2022, 7:32 PMwith Flow as flow:
for i in id_list:
dbt_run_function(i)
(I’m wondering if a simple for loop would achieve parallelism?)hieu le
07/20/2022, 9:11 AM