Marc Lipoff
05/25/2023, 6:45 PMRoss Leung
05/25/2023, 7:36 PM__post_init__
method and checks and initializes a self
parameter, the __post_init__
gets called AGAIN when the a task run gets created.Sean Conroy
05/25/2023, 9:19 PMFailed("Flow run encountered an exception. PrefectHTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/XYZ/workspaces/XYZ/flow_runs/XYZ>'\nResponse: {'detail': 'Orchestration API rate limit reached'}\nFor more information check: <https://httpstatuses.com/429>\n"
I don't know how this is possible - since I'm in no way interacting with prefect server. I've used previous versions of Prefect and never seen this before. I've replaced the specific strings with XYZRyan Morshead
05/25/2023, 9:24 PMException ignored in: <async_generator object add_event_loop_shutdown_callback.<locals>.on_shutdown at 0x7f3d81b374c0>
Traceback (most recent call last):
File "/home/jon/miniconda3/envs/abraxas-env/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 308, in on_shutdown
AttributeError: 'NoneType' object has no attribute 'pop'
Which seems related to this issue (which I’ve already commented on). I have no idea how to solve this. It seems like some odd threading/multiprocessing bug where EVENT_LOOP_GC_REFS is not being copied over correctly somehow.nicholasnet
05/25/2023, 9:45 PMdef child_flow_c():
d = run_deployment("child-flow-d/dep-child-d")
# How can I be sure that run_deployment("child-flow-d/dep-child-d") is completed before moving to next line/deployment
e = run_deployment("child-flow-d/dep-child-d")
How we check the status of run_deployment
is completed before executing another run_deployment
Jacob Bedard
05/25/2023, 9:45 PMJohn Horn
05/25/2023, 10:24 PMNico Neumann
05/26/2023, 1:05 AMprefect==2.10.11
version because of sqlalchemy 2.0 support. Thanks for the fast support there! 🚀
Now I have a problem with my custom block when I try to run my flows: KeyError: "No class found for dispatch key 'my-custom-block' in registry for type 'Block'."
prefect_aws==0.3.2
is installed and PREFECT_EXTRA_ENTRYPOINTS
is set. I’m using prefect cloud and the agent is updated to the latest version.
I downgraded my prefect docker image to version 2.10.10 and it works fine again. Full log attached in the threadNimesh Kumar
05/26/2023, 7:43 AMNikhil Joseph
05/26/2023, 8:17 AMOrchestration API rate limit reached
all day, any idea what's up?
I have mapped functions that around 3-4k iterations.
by the time I am around half of it. I hit this error and it crashes on its own.
When I retry, by the time it finds all completed tasks (checkpoints), it will raise the error again and won't move forward.
using prefect:2.10.10-python3.10 and running on ecs with s3 result storageDeceivious
05/26/2023, 8:26 AMGiacomo Chiarella
05/26/2023, 9:19 AMDeceivious
05/26/2023, 10:20 AMNikhil Joseph
05/26/2023, 11:43 AMJohn Mizerany
05/26/2023, 1:27 PMSimon
05/26/2023, 2:31 PMclass AnnotationAirtable(Base)
declaration with typing.@runtime_checkable, but received the same error.
I have tried separating the flow import with another module as a buffer (in desperation, to try to stop the ORM model import from happening!), also to no avail.massintha azamoum
05/26/2023, 3:11 PMMansour Zayer
05/26/2023, 5:27 PMChris Goddard
05/26/2023, 6:08 PMmain
function - with everything else being plain async python) and i’m still getting random hangs — and when I kill the script, it seems (as best as I can tell), that the KeyboardInterrupt
gets caught somewhere within the prefect codebase.
what is curious is that when I dropped the top-level @flow
altogether and just ran as a plain python script, it hasn’t stalled yet (will update if it does).
is it possible that there’s something going on with longer-running (5min+) async flows that’s causing them to hang at some point? I thought maybe it was to do with hitting the logging rate limit but by removing all of the task decorators I’ve managed to limit the overall prefect calls significantly.
I’m racking my brain at this point. anyone have any experience with long-running async programs? are there other factors that could be causing random hangs?jack
05/26/2023, 6:44 PMapi/work_queues/[ID]/status
endpoint, is it possible to modify the health_check_policy? Here is a typical response:
{
"healthy": true,
"late_runs_count": 0,
"last_polled": "2023-05-09T17:08:54.625892+00:00",
"health_check_policy": {
"maximum_late_runs": 0,
"maximum_seconds_since_last_polled": 60
}
}
Charles Leung
05/26/2023, 8:34 PMJenia Varavva
05/26/2023, 8:40 PMrun_deployment()
doesn't expose a PrefectFuture()
interface similar to Task.submit()
? For a flow like:
@flow
def f():
a = subflow_foo.submit(1) # .submit() doesn't exist on Flow or run_deployment().
b = subflow_foo.submit(2)
c = subflow_foo.submit(3)
x = subflow_bar(a, b)
y = subflow_bar(b, c)
x.wait()
y.wait()
I want to get the engine's dependency tracking to kick off downstream sub-flows as soon as their dependencies are satisfied. It feels like I could wrap run_deployment() in a future myself, but maybe I'm missing some non-obvious reason this wouldn't work?Jacob Bedard
05/26/2023, 9:24 PMFarhood Etaati
05/27/2023, 10:42 AMRobert Banick
05/29/2023, 3:11 AMFREQ=DAILY;BYHOUR=10;BYMINUTE=21
.
Even more curiously, sometimes the schedule will seem to work correctly immediately after running. Then I will check back some hours or a day later and the duplication will have happened.Nikhil Joseph
05/29/2023, 8:29 AMLuis Cebrián
05/29/2023, 10:30 AMThe notification block was invalid: KeyError(\"No class found for dispatch key 'slack-incoming-webhook' in registry for type 'Block'.\")
Is this a bug?Soami Charan
05/29/2023, 11:42 AMtask
with cache_key_fn
, then that where is that cache is stored in Prefect cloud?
If I use self hosted version then where this cache is stored and retrieved?Nicholas
05/29/2023, 2:04 PMtask
decorator is causing some issues with mypy
. 🧵Florent VanDeMoortele
05/29/2023, 4:03 PMprefect_docker
template to build a docker image and to push it on my private registry (on Gcloud) but it looks like it's not possible to use an other registry than docker. Any idea?
Otherwise, what's the best practice to deploy a docker image flow on a private storage?