Gabriel Milan
02/15/2022, 4:41 PMvalues.yaml
file such as
prefectConfig:
services:
towel:
max_scheduled_runs_per_flow: 50
and when I try to upgrade my deployments with this, the towel pod doesn't get redeployed and the issue is still there. Is this way correct or should I do anything else?Adi Gandra
02/15/2022, 5:13 PMXavier Babu
02/15/2022, 5:47 PMJake
02/15/2022, 6:59 PMConnectionClosedError('Connection was closed before we received a valid response from endpoint URL: "<insert_s3_url_here>.prefect_result".',)
It is causing our flows to fail intermittently and I’m not sure how to debug this.David Wang
02/15/2022, 8:24 PMKevin Mullins
02/15/2022, 8:34 PMPrefect Core encourages the use of small "helper" tasks. Won't those these inflate my billable usage?
Tasks that run for less than one second do not count as billable usage, so you can continue to design workflows that follow best practice and include as many helper tasks as you need.
Now that I’m using Prefect more and watching my task usage. I’ve noticed that both Secrets (EnvVarSecret in this case) which behind the scenes are a task and very simple tasks that just wrap parameters/task results into another structure vary quite a bit in runtime and often exceed the <1 second rule and thus will get billed. This is especially worrisome to me for secrets these tasks are necessary for securely handling things and some of our flows have quite a few secrets.
I’m not sure what a solution would be, but wanted to see if anybody has noticed this and if there could be anything done to ensure these simple tasks don’t inflate actual bills.Chowdary
02/15/2022, 10:04 PMChowdary
02/15/2022, 10:05 PMbrian
02/15/2022, 10:56 PM--param
to set parameters locally. I’ve just started working on a flow that will depend on a prefect secret, and would love to be able to set a value for the secret locally. Is there a way to do this? My naive hope was that prefect run
would have a --secret
flag similar to --param but this doesn’t seem to be the caseWilliam Grim
02/16/2022, 12:26 AMNitin Bansal
02/16/2022, 6:29 AMMukamisha jocelyne
02/16/2022, 8:45 AMNoah Holm
02/16/2022, 10:56 AMtask1
-> task2
-> task3
Task3 is the last task of the flow and has other upstream dependencies which might cause task3 to be skipped in a flow run. Today a flow run got the following states: Failed
-> TriggerFailed
-> Skipped
I assume the flow is considered successful since task3 was skipped and skipped states are considered successful (that makes sense). But we don’t want to consider a flow run successful if any tasks in the flow has a failed state. Refreshing my skills in the docs I’m thinking of handling it with state handlers, but I assume that a state handler on the flow would still see the flow state as successful due to the last skipped task. Is the only way forward here to have a state handler on each of the tasks in the flow?Chris Arderne
02/16/2022, 11:49 AMDaniel Nilsen
02/16/2022, 12:08 PMJason Motley
02/16/2022, 1:00 PMRichard Hughes
02/16/2022, 2:55 PMGuillaume Latour
02/16/2022, 3:04 PMTypeError: cannot pickle 'lxml.etree.XMLSchema' object
Of course the result of one of my task is of this type and I would prefer not to change it.
I tried to launch that process again with config.flows.checkpointing = "false"
in the ~/.prefect/config.toml
file but I got the same error...
This is an error that I cannot reproduce locally with prefect run <flow>
. Do you guys have any leads on how I can solve this issue or at least reproduce the error locally?Emma Rizzi
02/16/2022, 3:47 PMSean Talia
02/16/2022, 4:12 PMSam Werbalowsky
02/16/2022, 6:21 PMChris Reuter
02/16/2022, 7:43 PMJason Motley
02/16/2022, 8:11 PMSET procedure2 := bla bla
etc..Daniel Komisar
02/16/2022, 8:16 PMwhere
filter with a task id, but it works when I query by flow run id.Daniel Komisar
02/16/2022, 9:23 PMcase(False)
?Farid
02/16/2022, 9:32 PMResult
using Prefect? I dump the dict objects to str
JSONs using json.dumps
and then save them using S3Result
or LocalResult
and noticed both of them get some extra characters added to the beginning or end of json object:
��QXQ >> to the beginning
�. >> to the end
which makes it un-parseable.
Saving the same object using with open()
on local machine does not bear those extra charactersKelly Huang
02/17/2022, 1:37 AMAn Hoang
02/17/2022, 5:02 AMtask_B
) downstream of a mapped task (task_A
). Some of the input for task_B
(output of task_A
) is prefect.engine.signals.SKIP(None)
. In task_B
, do I have to filter out these skip signals? If output of task_B
is mapped to task_C
... task_X
, do I have to do if isinstance(input, prefect.signals.SKIP): raise prefect.signals.SKIP
for every single task in the chain?Ayah Safeen
02/17/2022, 8:46 AMdamien michelle
02/17/2022, 10:50 AM