Darragh
06/16/2020, 1:20 PMAn error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
Appears to be amazon complaining about the job family name, but I don’t know if there’s a way around it in the agent config.Zachary Hughes
06/16/2020, 1:32 PMDarragh
06/16/2020, 1:33 PMZachary Hughes
06/16/2020, 2:01 PMenable_task_revisions
might help alleviate some of this. It'll really depend on your use case
• Since this is essentially a rate-limiting issue, you could try limiting the number of connections by altering the botocore
config passed in (link below)
While this is an AWS issue, there's definitely room for us to handle this more gracefully. If you don't mind creating an issue, we can definitely look into how to improve the error handling here!
https://docs.prefect.io/orchestration/agents/fargate.html#configurationDarragh
06/16/2020, 2:24 PMZachary Hughes
06/16/2020, 2:30 PMBraun Reyes
06/16/2020, 2:41 PMBraun Reyes
06/16/2020, 2:42 PMBraun Reyes
06/16/2020, 2:44 PMDarragh
06/16/2020, 3:29 PMRegisterTaskDefinition
is what’s happening…Darragh
06/16/2020, 3:31 PMDarragh
06/16/2020, 3:36 PMBraun Reyes
06/16/2020, 3:39 PMBraun Reyes
06/16/2020, 3:40 PMDarragh
06/16/2020, 3:40 PMBraun Reyes
06/16/2020, 3:40 PMDarragh
06/16/2020, 3:40 PMBraun Reyes
06/16/2020, 3:40 PMBraun Reyes
06/16/2020, 3:41 PMDarragh
06/16/2020, 3:44 PMBraun Reyes
06/16/2020, 3:48 PMBraun Reyes
06/16/2020, 3:49 PMBraun Reyes
06/16/2020, 3:49 PMDarragh
06/16/2020, 3:50 PMDarragh
06/16/2020, 3:51 PMBraun Reyes
06/16/2020, 3:51 PMDarragh
06/16/2020, 3:51 PMBraun Reyes
06/16/2020, 3:52 PMBraun Reyes
06/16/2020, 3:52 PMDarragh
06/16/2020, 3:52 PMBraun Reyes
06/16/2020, 3:52 PMBraun Reyes
06/16/2020, 3:52 PMDarragh
06/16/2020, 3:53 PMDarragh
06/16/2020, 3:53 PMDarragh
06/16/2020, 3:54 PMjosh
06/16/2020, 3:58 PM[cloud.agent]
level = "DEBUG"
or as an env var:
export PREFECT__CLOUD__AGENT__LEVEL="DEBUG"
Darragh
06/16/2020, 4:05 PMBraun Reyes
06/16/2020, 4:05 PMBraun Reyes
06/16/2020, 4:06 PMDarragh
06/16/2020, 4:06 PMBraun Reyes
06/16/2020, 4:06 PMDarragh
06/16/2020, 4:08 PMBraun Reyes
06/16/2020, 4:09 PMBraun Reyes
06/16/2020, 4:09 PMDarragh
06/16/2020, 4:09 PMBraun Reyes
06/16/2020, 4:10 PMDarragh
06/16/2020, 4:10 PMDarragh
06/16/2020, 4:10 PMBraun Reyes
06/16/2020, 4:10 PMBraun Reyes
06/16/2020, 4:10 PMBraun Reyes
06/16/2020, 4:11 PMBraun Reyes
06/16/2020, 4:12 PMDarragh
06/16/2020, 4:13 PMBraun Reyes
06/16/2020, 4:14 PMDarragh
06/16/2020, 4:14 PMBraun Reyes
06/16/2020, 4:14 PMDarragh
06/16/2020, 4:16 PMDarragh
06/16/2020, 4:17 PMmax_pool_connections
is default 10, so I thought the current test case would be fine…..Braun Reyes
06/16/2020, 4:18 PMDarragh
06/16/2020, 4:20 PMBraun Reyes
06/16/2020, 4:20 PMBraun Reyes
06/16/2020, 4:20 PMBraun Reyes
06/16/2020, 4:20 PMDarragh
06/16/2020, 4:21 PMjosh
06/16/2020, 4:21 PMmax_workers = min(32, (os.cpu_count() or 1) + 4)
There’s an open issue here to change this behavior entirely https://github.com/PrefectHQ/prefect/issues/2468 if you’re interested in voicing your thoughtsDarragh
06/16/2020, 4:21 PMBraun Reyes
06/16/2020, 4:25 PMBraun Reyes
06/16/2020, 4:27 PMDarragh
06/16/2020, 4:28 PMDarragh
06/16/2020, 4:28 PMBraun Reyes
06/16/2020, 4:28 PMBraun Reyes
06/16/2020, 4:29 PMDarragh
06/16/2020, 4:29 PMDarragh
06/16/2020, 4:30 PMDarragh
06/16/2020, 4:30 PMDarragh
06/16/2020, 5:11 PMBraun Reyes
06/16/2020, 5:11 PMBraun Reyes
06/16/2020, 5:12 PMZachary Hughes
06/16/2020, 5:12 PMSubmitted
or beyond is an indication the work's been accepted.Braun Reyes
06/16/2020, 5:13 PMjosh
06/16/2020, 5:17 PMDarragh
06/16/2020, 5:20 PMZachary Hughes
06/16/2020, 5:37 PMstate = Submitted
would do the trick.Darragh
06/16/2020, 5:44 PMDarragh
06/16/2020, 5:46 PMZachary Hughes
06/16/2020, 5:50 PMDarragh
06/16/2020, 6:03 PMDarragh
06/16/2020, 6:06 PMZachary Hughes
06/16/2020, 6:11 PMDarragh
06/16/2020, 6:29 PMDarragh
06/24/2020, 4:59 PMBraun Reyes
06/24/2020, 5:29 PMDarragh
06/24/2020, 5:34 PMLukas
10/22/2020, 1:53 PMToo many concurrent attempts to create a new revision of the specified family.
error. Basically same context as for Darragh, I'm mapping over a list and create multiple FlowRunTasks. First time I run a new version of the fargate task I get this error. Did you guys find any solution for this issue? 🙂Braun Reyes
10/22/2020, 4:13 PMBraun Reyes
10/22/2020, 4:14 PMBraun Reyes
10/22/2020, 4:14 PMDarragh
10/22/2020, 4:16 PMBraun Reyes
10/26/2020, 5:52 PMBraun Reyes
10/26/2020, 6:05 PMRoey Brecher
11/20/2020, 6:55 AMRoey Brecher
11/20/2020, 6:57 AMenable_task_revisions
to False, would that actually solve the problem? (but create a “messy” Task definitions table in AWS 😂)
Another workaround I’m thinking of: When registering the flow, run a dummy instance of the flow that will force a new task revision to be registered on the spot.Roey Brecher
11/20/2020, 7:04 AMBraun Reyes
01/22/2021, 5:18 PMECSRun(
task_definition_arn=task_definition_arn, run_task_kwargs=dex_run_task_kwargs
)
Braun Reyes
01/22/2021, 5:19 PM