https://prefect.io
Join Slack
Can some one tell me why my flow is getting picked up by 2 agents?
# ask-community
d
Can some one tell me why my flow is getting picked up by 2 agents?
j
In a rare race condition, you might have the same flow run picked up by two identical agents, but the first flow run to start should cause the duplicate(s) to abort (race to be the first one to start executing, which is harmless (if confusing)). Your zoom-in is blocking the full logs, but it looks like you're actually seeing tasks execute once for each flow run, which should never happen. Is that accurate?
Are you using Cloud or Server?
d
Server
j
Can you post the full logs for that flow run?
d
I had this happen previously but i thought it was because I had 2 agents with the same name. but i have since change this to have uuid as names
The two agents are running all the tasks in parallel until the very end of the entire flow when one of them fails because the the flow state has already been updated by the other one
this is happening for all my flows
Copy code
agent:
        image: <http://259671471835.dkr.ecr.ap-southeast-2.amazonaws.com/nexu/scripts:latest|259671471835.dkr.ecr.ap-southeast-2.amazonaws.com/nexu/scripts:latest>
        entrypoint:
            - /bin/bash
            - -c
            - prefect agent local start --name $$(uuid) --import-path ~/scripts --show-flow-logs --log-level INFO --no-storage-labels -l s3-flow-storage -l gcs-flow-storage -l stl
        environment:
            - PREFECT__USER_CONFIG_PATH=/home/nexu/scripts/prefect/agent/config.toml
            - PREFECT__SERVER__HOST=<http://172.31.7.3>
            - PREFECT__SERVER__PORT=4200
            - BACKUP_DIR=/home/nexu/scripts/data/backup
        volumes:
            - nexu-data:/home/nexu/scripts/data
            - /dev/shm:/dev/shm
        restart: "always"
        logging:
            driver: awslogs
            options:
                awslogs-group: prefect-prod
                awslogs-region: ap-southeast-2
                awslogs-stream-prefix: agent
j
Hmmm, looking closer it looks like this limiting was never implemented in Prefect Server. So if you have multiple agents all matching on the same set of labels, you run the risk of a flow getting run multiple times.
Prefect Cloud will prevent this, but that feature was never ported over to Server.
d
lovely
j
For now, your best option is to avoid having agents with overlapping labels. Usually we recommend users deploy with a minimal number of agents and rely on the backing system for scalability. So for example, instead of deploying 100 local agents in individual containers, deploy one ECS agent that can kick off many concurrent flow runs in separate containers.
d
so... i either add labels or contribute the equivalent code to server or hope the code gets pushed upstream from cloud
j
This is definitely fixable though, just not currently fixed.
d
thanks
j
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by