Question, if I have multiple agents with the same ...
# ask-communitya
Adam Brusselback
03/09/2021, 3:45 PMQuestion, if I have multiple agents with the same labels, they seem to be running the tasks in a flow once each...that seems wrong
j
Jim Crist-Harif
03/09/2021, 3:46 PMAre you using prefect server or prefect cloud?
a
Adam Brusselback
03/09/2021, 3:46 PMserver
Adam Brusselback
03/09/2021, 3:46 PMj
Jim Crist-Harif
03/09/2021, 3:47 PMThis is a known issue with prefect server. https://github.com/PrefectHQ/server/issues/203
Jim Crist-Harif
03/09/2021, 3:47 PMPrefect cloud includes a lock preventing multiple agents from submitting the same flow run, prefect server currently has no such lock. We currently recommend prefect server users to not have agents with intersecting labels.
Jim Crist-Harif
03/09/2021, 3:49 PMSince most prefect users rely on a scalable deployment backend (e.g. k8s, ecs, etc...) rather than scaling up the number of agents, in practice this issue hasn't come up much, but is definitely an issue we want to fix.
a
Adam Brusselback
03/09/2021, 3:50 PMGotcha. I know timeframes on these types of things are...inaccurate? just wondering if there is a plan on porting that back?
j
Jim Crist-Harif
03/09/2021, 3:51 PMI'm not sure what the timeframe on this would be. I consider this a fairly serious issue, so hopefully this would be in the next couple weeks.
a
Adam Brusselback
03/09/2021, 3:53 PMOkay, cool so that is very reasonable. Just wanting to figure out if I should make architecture changes or what, but we're still in testing. Waiting a couple weeks for a fix would be far easier
j
Jim Crist-Harif
03/09/2021, 3:54 PMNo promises on it being a couple weeks, but it's not something that's fundamentally wrong with prefect server, so it can/will be fixed eventually.
a
Adam Brusselback
03/09/2021, 3:54 PMYeah, makes sense and totally understood.
Adam Brusselback
03/09/2021, 3:55 PMIs the cloud versions feature that takes care of that responsible for more than just that, which is why it wasn't ported? if so what else is that if you don't mind.\
j
Jim Crist-Harif
03/09/2021, 3:55 PMNo, it makes use of redis, which we don't use in prefect server. I think we can implement an equivalent feature without redis though.
a
Adam Brusselback
03/09/2021, 3:58 PMAh, got it. Probably can get by with just the locking features in Postgres for regular sized deployments without any performance issues, but since I don't know the exact use case it's hard to tell.
j
Jim Crist-Harif
03/09/2021, 3:59 PMYeah, prefect cloud has to scale much more than we expect a typical prefect server deployment to.
a
Adam Brusselback
03/09/2021, 3:59 PM(i've had pretty good luck using PG's advisory locks for a couple things which could have brought in a whole different stack otherwise)
j
Jim Crist-Harif
03/09/2021, 4:00 PMAnyway, I suggest you follow the linked github issue for updates, and maybe consider working around this by deploying agents with non-intersecting labels for now.
3 Views