Adam Brusselback

    Adam Brusselback

    1 year ago
    Question, if I have multiple agents with the same labels, they seem to be running the tasks in a flow once each...that seems wrong
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Are you using prefect server or prefect cloud?
    Adam Brusselback

    Adam Brusselback

    1 year ago
    server
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    This is a known issue with prefect server. https://github.com/PrefectHQ/server/issues/203
    Prefect cloud includes a lock preventing multiple agents from submitting the same flow run, prefect server currently has no such lock. We currently recommend prefect server users to not have agents with intersecting labels.
    Since most prefect users rely on a scalable deployment backend (e.g. k8s, ecs, etc...) rather than scaling up the number of agents, in practice this issue hasn't come up much, but is definitely an issue we want to fix.
    Adam Brusselback

    Adam Brusselback

    1 year ago
    Gotcha. I know timeframes on these types of things are...inaccurate? just wondering if there is a plan on porting that back?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    I'm not sure what the timeframe on this would be. I consider this a fairly serious issue, so hopefully this would be in the next couple weeks.
    Adam Brusselback

    Adam Brusselback

    1 year ago
    Okay, cool so that is very reasonable. Just wanting to figure out if I should make architecture changes or what, but we're still in testing. Waiting a couple weeks for a fix would be far easier
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    No promises on it being a couple weeks, but it's not something that's fundamentally wrong with prefect server, so it can/will be fixed eventually.
    Adam Brusselback

    Adam Brusselback

    1 year ago
    Yeah, makes sense and totally understood.
    Is the cloud versions feature that takes care of that responsible for more than just that, which is why it wasn't ported? if so what else is that if you don't mind.\
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    No, it makes use of redis, which we don't use in prefect server. I think we can implement an equivalent feature without redis though.
    Adam Brusselback

    Adam Brusselback

    1 year ago
    Ah, got it. Probably can get by with just the locking features in Postgres for regular sized deployments without any performance issues, but since I don't know the exact use case it's hard to tell.
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Yeah, prefect cloud has to scale much more than we expect a typical prefect server deployment to.
    Adam Brusselback

    Adam Brusselback

    1 year ago
    (i've had pretty good luck using PG's advisory locks for a couple things which could have brought in a whole different stack otherwise)
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Anyway, I suggest you follow the linked github issue for updates, and maybe consider working around this by deploying agents with non-intersecting labels for now.