https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm having a weird mismatch between t...
# ask-marvin
k
@Marvin I'm having a weird mismatch between the CLI and webapp in the the work pool, in which my worflows never run, but I can't find any error or log:
Copy code
(py310)  ✘  /workspace  ↱ 815-poc-workflow-orchestration ±  export PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL="DEBUG"
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  export PREFECT_API_URL=<http://127.0.0.1:4200/api>
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect work-pool create fraio -t docker --overwrite               
Created work pool 'fraio'!

To start a worker for this work pool, run:

        prefect worker start --pool fraio

└── UUID: efe13a52-193e-4181-b43e-0fec28c6f9e8
└── Type: docker
└── Description: None
└── Status: Not ready
└── URL: <http://127.0.0.1:4200/work-pools/work-pool/fraio>

(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect worker start --pool fraio --work-queue default 

Discovered type 'docker' for work pool 'fraio'.
Worker 'DockerWorker 288f4652-24e8-40b0-b40c-2b6d1d773a13' started!
I can add deployments:
Copy code
$ python docker-deploy.py 
Successfully created/updated all deployments!

                   Deployments                    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Name                       ┃ Status  ┃ Details ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ trigger-flow/fraio-trigger │ applied │         │
└────────────────────────────┴─────────┴─────────┘

To schedule a run for this deployment, use the following command:

        $ prefect deployment run 'trigger-flow/fraio-trigger'


You can also run your flow via the Prefect UI: <http://127.0.0.1:4200/deployments/deployment/bdbbb4c5-1672-4fb0-84e1-8bc420d533f0>

Created/updated deployment: bdbbb4c5-1672-4fb0-84e1-8bc420d533f0
So far, so good, but any trigger get's stuk: it is added, but runs 
late
(never executes)
m
✅ thought for 640.4 seconds
Thanks for the details — you’ve got most pieces in place. When runs sit in Late, it almost always means the worker that’s polling your pool/queue isn’t seeing those runs or can’t start them. Let’s quickly verify the usual culprits. Quick checks to pinpoint the mismatch Please paste the output of these so we can zero in: - Verify your Prefect version and config are pointed at the same API as the UI
Copy code
prefect version
prefect config view --hide-defaults --show-sources
- Confirm the pool and queues exist as expected and are “ready”
Copy code
prefect work-pool ls -v
prefect work-pool inspect fraio
prefect work-queue ls -p fraio -v
prefect work-queue inspect default -p fraio
- Make sure your deployment is actually assigned to pool fraio and queue default
Copy code
prefect deployment inspect "trigger-flow/fraio-trigger"
- See the Late runs the scheduler created
Copy code
prefect flow-run ls --state LATE --limit 20
What commonly causes Late runs in this setup - Deployment is pointing at a different work pool or queue than the worker is polling. - The work pool is paused or has a concurrency limit of 0. - The worker is connected to a different API than the UI/CLI you used to create the deployment. - Worker started fine but cannot start Docker jobs (e.g., no access to the Docker daemon) — the worker logs would show Docker client errors. - Networking/API URL problems for containers (relevant after pickup, but worth fixing early). How to validate/fix quickly 1) Ensure deployment targets the same pool and queue the worker is polling If the deployment isn’t on pool fraio and queue default, redeploy with explicit settings. In Prefect 3.x, with Flow.deploy:
Copy code
flow.deploy(
  name="fraio-trigger",
  work_pool_name="fraio",
  work_queue_name="default",
  # image=..., build/push if needed
)
2) Verify the worker is visible in the UI on the fraio pool Open your pool page and check if a worker appears. If it does not, your worker is probably talking to a different API. The worker start output
Copy code
prefect worker start --pool fraio --work-queue default
should show polling logs shortly after you create/schedule a run. If you see nothing, it’s likely not seeing the runs. 3) Turn up worker/client logs for more signal Run the worker with:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_WORKERS_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_CLIENT_LEVEL=DEBUG
prefect worker start --pool fraio --work-queue default
You should see lines like “Submitting flow run…” or any Docker errors. 4) If you’re running the worker inside a container A Docker-type worker must be able to reach the Docker daemon. If your worker is inside a dev container, make sure /var/run/docker.sock is mounted or that it can reach a Docker endpoint. Otherwise, starting flow containers will fail. 5) Prep for container networking (once pickup works) Since your PREFECT_API_URL is http://127.0.0.1:4200/api, containers started by the Docker worker won’t be able to reach 127.0.0.1 on the host. Use a host-reachable address: - Set PREFECT_API_URL in your deployment/job variables to something like http://host.docker.internal:4200/api (on Mac/Windows) or your host IP (Linux). Docs: - Workers concept - Work queue CLI - Running Prefect Server and networking tips If you share the outputs of the inspect/ls commands above (especially the deployment inspect showing work_pool_name/work_queue_name), I can pinpoint why those runs are going Late and provide an exact fix.
k
@Marvin
Copy code
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect version
prefect config view --hide-defaults --show-sources
Version:              3.4.21
API version:          0.8.4
Python version:       3.10.12
Git commit:           4fa9d9b4
Built:                Thu, Oct 02, 2025 07:14 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          server
Pydantic version:     2.11.9
Server:
  Database:           sqlite
  SQLite version:     3.37.2
Integrations:
  prefect-kubernetes: 0.6.5
  prefect-docker:     0.6.6
🚀 you are connected to:
<http://127.0.0.1:4200>
PREFECT_PROFILE='ephemeral'
PREFECT_API_URL='<http://127.0.0.1:4200/api>' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect work-pool ls -v
prefect work-pool inspect fraio
prefect work-queue ls -p fraio -v
prefect work-queue inspect default -p fraio
                                                                                                              Work Pools                                                                                                               
┏┳┳━━━━━━━━━━━━━━━━━━━┳┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃┃┃                ID ┃┃ Base Job Template                                                                                                                                                                                            ┃
┡╇╇━━━━━━━━━━━━━━━━━━━╇╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│││ 3a3d730e-86d9-49… ││ {'variables': {'type': 'object', '$defs': {'DockerRegistryCredentials': {'type': 'object', 'title': 'DockerRegistryCredentials', 'required': ['username', 'password', 'registry_url'], 'properties': {'reau… │
└┴┴───────────────────┴┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                      (**) denotes a paused pool                                                                                                       
WorkPool(
    id='3a3d730e-86d9-498b-a545-7880592d5b9f',
    created=DateTime(2025, 10, 4, 8, 3, 9, 206793, tzinfo=Timezone('UTC')),
    updated=DateTime(2025, 10, 4, 8, 4, 25, 245636, tzinfo=Timezone('UTC')),
    name='fraio',
    type='docker',
    base_job_template={
        'variables': {
            'type': 'object',
            '$defs': {
                'DockerRegistryCredentials': {
                    'type': 'object',
                    'title': 'DockerRegistryCredentials',
                    'required': ['username', 'password', 'registry_url'],
                    'properties': {
                        'reauth': {'type': 'boolean', 'title': 'Reauth', 'default': True, 'description': 'Whether or not to reauthenticate on each interaction.'},
                        'password': {'type': 'string', 'title': 'Password', 'format': 'password', 'writeOnly': True, 'description': 'The password to log into the registry with.'},
                        'username': {'type': 'string', 'title': 'Username', 'description': 'The username to log into the registry with.'},
                        'registry_url': {'type': 'string', 'title': 'Registry Url', 'examples': ['index.docker.io'], 'description': 'The URL to the registry. Generally, "http" or "https" can be omitted.'}
                    },
                    'description': 'Store credentials for interacting with a Docker Registry.',
                    'secret_fields': ['password'],
                    'block_type_slug': 'docker-registry-credentials',
                    'additionalProperties': True,
                    'block_schema_references': {}
                }
            },
            'properties': {
                'env': {'type': 'object', 'title': 'Environment Variables', 'description': 'Environment variables to set when starting a flow run.', 'additionalProperties': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}},
                'name': {'anyOf': [{'type': 'string'}, {'type': 'null'}], 'title': 'Name', 'default': None, 'description': 'Name given to infrastructure created by the worker using this job configuration.'},
                'image': {
                    'type': 'string',
                    'title': 'Image',
                    'examples': ['docker.io/prefecthq/prefect:3-latest'],
                    'description': 'The image reference of a container image to use for created jobs. If not set, the latest Prefect image will be used.'
                },
                'labels': {'type': 'object', 'title': 'Labels', 'description': 'Labels applied to infrastructure created by the worker using this job configuration.', 'additionalProperties': {'type': 'string'}},
                'command': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Command',
                    'default': None,
                    'description': 'The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.'
                },
                'volumes': {'type': 'array', 'items': {'type': 'string'}, 'title': 'Volumes', 'examples': ['/my/local/path:/path/in/container'], 'description': 'A list of volume to mount into created containers.'},
                'networks': {'type': 'array', 'items': {'type': 'string'}, 'title': 'Networks', 'description': 'Docker networks that created containers should be connected to.'},
                'mem_limit': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Memory Limit',
                    'default': None,
                    'description': 'Memory limit of created containers. Accepts a value with a unit identifier (e.g. 100000b, 1000k, 128m, 1g.) If a value is given without a unit, bytes are assumed.'
                },
                'privileged': {'type': 'boolean', 'title': 'Privileged', 'default': False, 'description': 'Give extended privileges to created container.'},
                'auto_remove': {'type': 'boolean', 'title': 'Auto Remove', 'default': False, 'description': 'If set, containers will be deleted on completion.'},
                'network_mode': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Network Mode',
                    'default': None,
                    'description': "The network mode for the created containers (e.g. host, bridge). If 'networks' is set, this cannot be set."
                },
                'memswap_limit': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Memory Swap Limit',
                    'default': None,
                    'description': 'Total memory (memory + swap), -1 to disable swap. Should only be set if `mem_limit` is also set. If `mem_limit` is set, this defaults toallowing the container to use as much swap as memory. For 
example, if `mem_limit` is 300m and `memswap_limit` is not set, containers can use 600m in total of memory and swap.'
                },
                'stream_output': {'type': 'boolean', 'title': 'Stream Output', 'default': True, 'description': 'If set, the output from created containers will be streamed to local standard output.'},
                'image_pull_policy': {
                    'anyOf': [{'enum': ['IfNotPresent', 'Always', 'Never'], 'type': 'string'}, {'type': 'null'}],
                    'title': 'Image Pull Policy',
                    'default': None,
                    'description': 'The image pull policy to use when pulling images.'
                },
                'registry_credentials': {'anyOf': [{'$ref': '#/$defs/DockerRegistryCredentials'}, {'type': 'null'}], 'default': None, 'description': 'Credentials for logging into a Docker registry to pull images from.'},
                'container_create_kwargs': {
                    'anyOf': [{'type': 'object', 'additionalProperties': True}, {'type': 'null'}],
                    'title': 'Container Configuration',
                    'default': None,
                    'description': 'Configuration for containers created by workers. See the [`docker-py` documentation](<https://docker-py.readthedocs.io/en/stable/containers.html>) for accepted values.'
                }
            },
            'description': "..."
        },
        'job_configuration': {
            'env': '{{ env }}',
            'name': '{{ name }}',
            'image': '{{ image }}',
            'labels': '{{ labels }}',
            'command': '{{ command }}',
            'volumes': '{{ volumes }}',
            'networks': '{{ networks }}',
            'mem_limit': '{{ mem_limit }}',
            'privileged': '{{ privileged }}',
            'auto_remove': '{{ auto_remove }}',
            'network_mode': '{{ network_mode }}',
            'memswap_limit': '{{ memswap_limit }}',
            'stream_output': '{{ stream_output }}',
            'image_pull_policy': '{{ image_pull_policy }}',
            'registry_credentials': '{{ registry_credentials }}',
            'container_create_kwargs': '{{ container_create_kwargs }}'
        }
    },
    status=WorkPoolStatus.READY,
    storage_configuration=WorkPoolStorageConfiguration(),
    default_queue_id='834115d9-7a00-49d7-9c6a-49c1be0d2098'
)
                     Work Queues in Work Pool 'fraio'                      
┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Priority ┃ Concurrency Limit ┃ Description                    ┃
┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ default │ 1        │ None              │ The work pool's default queue. │
└─────────┴──────────┴───────────────────┴────────────────────────────────┘
                        (**) denotes a paused queue                        
WorkQueue(
    id='834115d9-7a00-49d7-9c6a-49c1be0d2098',
    created=DateTime(2025, 10, 4, 8, 3, 9, 210316, tzinfo=Timezone('UTC')),
    updated=DateTime(2025, 10, 4, 8, 18, 39, 659032, tzinfo=Timezone('UTC')),
    name='default',
    description="The work pool's default queue.",
    work_pool_name='fraio',
    work_pool_id=UUID('3a3d730e-86d9-498b-a545-7880592d5b9f'),
    last_polled=DateTime(2025, 10, 4, 8, 18, 39, 658388, tzinfo=Timezone('UTC')),
    status=WorkQueueStatus.READY
)
WorkQueueStatusDetail(healthy=False, late_runs_count=1, last_polled=DateTime(2025, 10, 4, 8, 18, 39, 658388, tzinfo=Timezone('UTC')), health_check_policy=WorkQueueHealthPolicy())
Work queue status issue? btw, I'd expect to run on a postgres, but it shows sq lite?
m
Your message was too long, here's your message at the allowed limit: ``` (py310) /workspace  ↱ 815-poc-workflow-orchestration ±  prefect version prefect config view --hide-defaults --show-sources Version 3 4 21 API version 0 8 4 Python version 3 10 12 Git commit 4fa9d9b4 Built Thu Oct 02 2025 07 14 PM OS/Arch linux/x86_64 Profile ephemeral Server type server Pydantic version 2 11 9 Server Database sqlite SQLite version 3 37 2 Integrations prefect-kubernetes 0 6 5 prefect-docker 0 6 6 🚀 you are connected to <http //127 0 0 1 4200> PREFECT_PROFILE='ephemeral' PREFECT_API_URL='<http //127 0 0 1 4200/api>' (from env) PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile) (py310) /workspace  ↱ 815-poc-workflow-orchestration ±  prefect work-pool ls -v prefect work-pool inspect fraio prefect work-queue ls -p fraio -v prefect work-queue inspect default -p fraio Work Pools ┏┳┳━━━━━━━━━━━━━━━━━━━┳┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃┃┃ ID ┃┃ Base Job Template ┃ ┡╇╇━━━━━━━━━━━━━━━━━━━╇╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │││ 3a3d730e-86d9-49… ││ {'variables' {'type' 'object' '$defs' {'DockerRegistryCredentials' {'type' 'object' 'title' 'DockerRegistryCredentials' 'required' ['username' 'password' 'registry_url'] 'properties' {'reau… │ └┴┴───────────────────┴┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ (**) denotes a paused pool WorkPool( id='3a3d730e-86d9-498b-a545-7880592d5b9f' created=DateTime(2025 10 4 8 3 9 206793 tzinfo=Timezone('UTC')) updated=DateTime(2025 10 4 8 4 25 245636 tzinfo=Timezone('UTC')) name='fraio' type='docker' base_job_template={ 'variables' { 'type' 'object' '$defs' { 'DockerRegistryCredentials' { 'type' 'object' 'title' 'DockerRegistryCredentials' 'required' ['username' 'password' 'registry_url'] 'properties' { 'reauth' {'type' 'boolean' 'title' 'Reauth' 'default' True 'description' 'Whether or not to reauthenticate on each interaction '} 'password' {'type' 'string' 'title' 'Password' 'format' 'password' 'writeOnly' True 'description' 'The password to log into the registry with '} 'username' {'type' 'string' 'title' 'Username' 'description' 'The username to log into the registry with '} 'registry_url' {'type' 'string' 'title' 'Registry Url' 'examples' ['index docker io'] 'description' 'The URL to the registry Generally http or https can be omitted '} } 'description' 'Store credentials for interacting with a Docker Registry ' 'secret_fields' ['password'] 'block_type_slug' 'docker-registry-credentials' 'additionalProperties' True 'block_schema_references' {} } } 'properties' { 'env' {'type' 'object' 'title' 'Environment Variables' 'description' 'Environment variables to set when starting a flow run ' 'additionalProperties' {'anyOf' [{'type' 'string'} {'type' 'null'}]}} 'name' {'anyOf' [{'type' 'string'} {'type' 'null'}] 'title' 'Name' 'default' None 'description' 'Name given to infrastructure created by the worker using this job configuration '} 'image' { 'type' 'string' 'title' 'Image' 'examples' ['docker io/prefecthq/prefect 3-latest'] 'description' 'The image reference of a container image to use for created jobs If not set the latest Prefect image will be used ' } 'labels' {'type' 'object' 'title' 'Labels' 'description' 'Labels applied to infrastructure created by the worker using this job configuration ' 'additionalProperties' {'type' 'string'}} 'command' { 'anyOf' [{'type' 'string'} {'type' 'null'}] 'title' 'Command' 'default' None 'description' 'The command to use when starting a flow run In most cases
this should be left blank and the command will be automatically generated by the worker ' } 'volumes' {'type' 'array' 'items' {'type' 'string'} 'title' 'Volumes' 'examples' ['/my/local/path /path/in/container'] 'description' 'A list of volume to mount into created containers '} 'networks' {'type' 'array' 'items' {'type' 'string'} 'title' 'Networks' 'description' 'Docker networks that created containers should be connected to '} 'mem_limit' { 'anyOf' [{'type' 'string'} {'type' 'null'}] 'title' 'Memory Limit' 'default' None 'description' 'Memory limit of created containers Accepts a value with
k
@Marvin
Copy code
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect version
prefect config view --hide-defaults --show-sources
Version:              3.4.21
API version:          0.8.4
Python version:       3.10.12
Git commit:           4fa9d9b4
Built:                Thu, Oct 02, 2025 07:14 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          server
Pydantic version:     2.11.9
Server:
  Database:           sqlite
  SQLite version:     3.37.2
Integrations:
  prefect-kubernetes: 0.6.5
  prefect-docker:     0.6.6
🚀 you are connected to:
<http://127.0.0.1:4200>
PREFECT_PROFILE='ephemeral'
PREFECT_API_URL='<http://127.0.0.1:4200/api>' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect work-pool ls -v
prefect work-pool inspect fraio
prefect work-queue ls -p fraio -v
prefect work-queue inspect default -p fraio
                                                                                                              Work Pools                                                                                                               
┏┳┳━━━━━━━━━━━━━━━━━━━┳┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃┃┃                ID ┃┃ Base Job Template                                                                                                                                                                                            ┃
┡╇╇━━━━━━━━━━━━━━━━━━━╇╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│││ 3a3d730e-86d9-49… ││ {'variables': {'type': 'object', '$defs': {'DockerRegistryCredentials': {'type': 'object', 'title': 'DockerRegistryCredentials', 'required': ['username', 'password', 'registry_url'], 'properties': {'reau… │
└┴┴───────────────────┴┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                      (**) denotes a paused pool                                                                                                       
WorkPool(
    id='3a3d730e-86d9-498b-a545-7880592d5b9f',
    created=DateTime(2025, 10, 4, 8, 3, 9, 206793, tzinfo=Timezone('UTC')),
    updated=DateTime(2025, 10, 4, 8, 4, 25, 245636, tzinfo=Timezone('UTC')),
    name='fraio',
    type='docker',
    base_job_template={
        'variables': {
            'type': 'object',
            '$defs': {
                'DockerRegistryCredentials': {
                    'type': 'object',
                    'title': 'DockerRegistryCredentials',
                    'required': ['username', 'password', 'registry_url'],
                    'properties': {
                        'reauth': {'type': 'boolean', 'title': 'Reauth', 'default': True, 'description': 'Whether or not to reauthenticate on each interaction.'},
                        'password': {'type': 'string', 'title': 'Password', 'format': 'password', 'writeOnly': True, 'description': 'The password to log into the registry with.'},
                        'username': {'type': 'string', 'title': 'Username', 'description': 'The username to log into the registry with.'},
                        'registry_url': {'type': 'string', 'title': 'Registry Url', 'examples': ['<http://index.docker.io|index.docker.io>'], 'description': 'The URL to the registry. Generally, "http" or "https" can be omitted.'}
                    },
                    'description': 'Store credentials for interacting with a Docker Registry.',
                    'secret_fields': ['password'],
                    'block_type_slug': 'docker-registry-credentials',
                    'additionalProperties': True,
                    'block_schema_references': {}
                }
            },
            'properties': {
                'env': {'type': 'object', 'title': 'Environment Variables', 'description': 'Environment variables to set when starting a flow run.', 'additionalProperties': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}},
                'name': {'anyOf': [{'type': 'string'}, {'type': 'null'}], 'title': 'Name', 'default': None, 'description': 'Name given to infrastructure created by the worker using this job configuration.'},
                'image': {
                    'type': 'string',
                    'title': 'Image',
                    'examples': ['<http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>'],
                    'description': 'The image reference of a container image to use for created jobs. If not set, the latest Prefect image will be used.'
                },
                'labels': {'type': 'object', 'title': 'Labels', 'description': 'Labels applied to infrastructure created by the worker using this job configuration.', 'additionalProperties': {'type': 'string'}},
                'command': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Command',
                    'default': None,
                    'description': 'The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.'
                },
                'volumes': {'type': 'array', 'items': {'type': 'string'}, 'title': 'Volumes', 'examples': ['/my/local/path:/path/in/container'], 'description': 'A list of volume to mount into created containers.'},
                'networks': {'type': 'array', 'items': {'type': 'string'}, 'title': 'Networks', 'description': 'Docker networks that created containers should be connected to.'},
                'mem_limit': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Memory Limit',
                    'default': None,
                    'description': '...'
                },
                'privileged': {'type': 'boolean', 'title': 'Privileged', 'default': False, 'description': 'Give extended privileges to created container.'},
                'auto_remove': {'type': 'boolean', 'title': 'Auto Remove', 'default': False, 'description': 'If set, containers will be deleted on completion.'},
                'network_mode': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Network Mode',
                    'default': None,
                    'description': "The network mode for the created containers (e.g. host, bridge). If 'networks' is set, this cannot be set."
                },
                'memswap_limit': {
                    'anyOf': [{'type': 'string'}, {'type': 'null'}],
                    'title': 'Memory Swap Limit',
                    'default': None,
                    'description': 'Total memory (memory + swap), -1 to disable swap. Should only be set if `mem_limit` is also set. If `mem_limit` is set, this defaults toallowing the container to use as much swap as memory. For 
example, if `mem_limit` is 300m and `memswap_limit` is not set, containers can use 600m in total of memory and swap.'
                },
                'stream_output': {'type': 'boolean', 'title': 'Stream Output', 'default': True, 'description': 'If set, the output from created containers will be streamed to local standard output.'},
                'image_pull_policy': {
                    'anyOf': [{'enum': ['IfNotPresent', 'Always', 'Never'], 'type': 'string'}, {'type': 'null'}],
                    'title': 'Image Pull Policy',
                    'default': None,
                    'description': 'The image pull policy to use when pulling images.'
                },
                'registry_credentials': {'anyOf': [{'$ref': '#/$defs/DockerRegistryCredentials'}, {'type': 'null'}], 'default': None, 'description': 'Credentials for logging into a Docker registry to pull images from.'},
                'container_create_kwargs': {
                    'anyOf': [{'type': 'object', 'additionalProperties': True}, {'type': 'null'}],
                    'title': 'Container Configuration',
                    'default': None,
                    'description': 'Configuration for containers created by workers. See the [`docker-py` documentation](<https://docker-py.readthedocs.io/en/stable/containers.html>) for accepted values.'
                }
            },
            'description': "..."
        },
        'job_configuration': {
            'env': '{{ env }}',
            'name': '{{ name }}',
            'image': '{{ image }}',
            'labels': '{{ labels }}',
            'command': '{{ command }}',
            'volumes': '{{ volumes }}',
            'networks': '{{ networks }}',
            'mem_limit': '{{ mem_limit }}',
            'privileged': '{{ privileged }}',
            'auto_remove': '{{ auto_remove }}',
            'network_mode': '{{ network_mode }}',
            'memswap_limit': '{{ memswap_limit }}',
            'stream_output': '{{ stream_output }}',
            'image_pull_policy': '{{ image_pull_policy }}',
            'registry_credentials': '{{ registry_credentials }}',
            'container_create_kwargs': '{{ container_create_kwargs }}'
        }
    },
    status=WorkPoolStatus.READY,
    storage_configuration=WorkPoolStorageConfiguration(),
    default_queue_id='834115d9-7a00-49d7-9c6a-49c1be0d2098'
)
                     Work Queues in Work Pool 'fraio'                      
┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Priority ┃ Concurrency Limit ┃ Description                    ┃
┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ default │ 1        │ None              │ The work pool's default queue. │
└─────────┴──────────┴───────────────────┴────────────────────────────────┘
                        (**) denotes a paused queue                        
WorkQueue(
    id='834115d9-7a00-49d7-9c6a-49c1be0d2098',
    created=DateTime(2025, 10, 4, 8, 3, 9, 210316, tzinfo=Timezone('UTC')),
    updated=DateTime(2025, 10, 4, 8, 18, 39, 659032, tzinfo=Timezone('UTC')),
    name='default',
    description="The work pool's default queue.",
    work_pool_name='fraio',
    work_pool_id=UUID('3a3d730e-86d9-498b-a545-7880592d5b9f'),
    last_polled=DateTime(2025, 10, 4, 8, 18, 39, 658388, tzinfo=Timezone('UTC')),
    status=WorkQueueStatus.READY
)
WorkQueueStatusDetail(healthy=False, late_runs_count=1, last_polled=DateTime(2025, 10, 4, 8, 18, 39, 658388, tzinfo=Timezone('UTC')), health_check_policy=WorkQueueHealthPolicy())
m
Your message was too long, here's your message at the allowed limit: ```(py310) /workspace  ↱ 815-poc-workflow-orchestration ±  prefect version prefect config view --hide-defaults --show-sources Version 3 4 21 API version 0 8 4 Python version 3 10 12 Git commit 4fa9d9b4 Built Thu Oct 02 2025 07 14 PM OS/Arch linux/x86_64 Profile ephemeral Server type server Pydantic version 2 11 9 Server Database sqlite SQLite version 3 37 2 Integrations prefect-kubernetes 0 6 5 prefect-docker 0 6 6 🚀 you are connected to <http //127 0 0 1 4200> PREFECT_PROFILE='ephemeral' PREFECT_API_URL='<http //127 0 0 1 4200/api>' (from env) PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile) (py310) /workspace  ↱ 815-poc-workflow-orchestration ±  prefect work-pool ls -v prefect work-pool inspect fraio prefect work-queue ls -p fraio -v prefect work-queue inspect default -p fraio Work Pools ┏┳┳━━━━━━━━━━━━━━━━━━━┳┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃┃┃ ID ┃┃ Base Job Template ┃ ┡╇╇━━━━━━━━━━━━━━━━━━━╇╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │││ 3a3d730e-86d9-49… ││ {'variables' {'type' 'object' '$defs' {'DockerRegistryCredentials' {'type' 'object' 'title' 'DockerRegistryCredentials' 'required' ['username' 'password' 'registry_url'] 'properties' {'reau… │ └┴┴───────────────────┴┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ (**) denotes a paused pool WorkPool( id='3a3d730e-86d9-498b-a545-7880592d5b9f' created=DateTime(2025 10 4 8 3 9 206793 tzinfo=Timezone('UTC')) updated=DateTime(2025 10 4 8 4 25 245636 tzinfo=Timezone('UTC')) name='fraio' type='docker' base_job_template={ 'variables' { 'type' 'object' '$defs' { 'DockerRegistryCredentials' { 'type' 'object' 'title' 'DockerRegistryCredentials' 'required' ['username' 'password' 'registry_url'] 'properties' { 'reauth' {'type' 'boolean' 'title' 'Reauth' 'default' True 'description' 'Whether or not to reauthenticate on each interaction '} 'password' {'type' 'string' 'title' 'Password' 'format' 'password' 'writeOnly' True 'description' 'The password to log into the registry with '} 'username' {'type' 'string' 'title' 'Username' 'description' 'The username to log into the registry with '} 'registry_url' {'type' 'string' 'title' 'Registry Url' 'examples' ['<http //index docker io|index docker io>'] 'description' 'The URL to the registry Generally http or https can be omitted '} } 'description' 'Store credentials for interacting with a Docker Registry ' 'secret_fields' ['password'] 'block_type_slug' 'docker-registry-credentials' 'additionalProperties' True 'block_schema_references' {} } } 'properties' { 'env' {'type' 'object' 'title' 'Environment Variables' 'description' 'Environment variables to set when starting a flow run ' 'additionalProperties' {'anyOf' [{'type' 'string'} {'type' 'null'}]}} 'name' {'anyOf' [{'type' 'string'} {'type' 'null'}] 'title' 'Name' 'default' None 'description' 'Name given to infrastructure created by the worker using this job configuration '} 'image' { 'type' 'string' 'title' 'Image' 'examples' ['<http //docker io/prefecthq/prefect 3-latest|docker io/prefecthq/prefect 3-latest>'] 'description' 'The image reference of a container image to use for created jobs If not set the latest Prefect image will be used ' } 'labels' {'type' 'object' 'title' 'Labels' 'description' 'Labels applied to infrastructure created by the worker using this job configuration ' 'additionalProperties' {'type' 'string'}} 'command' { 'anyOf' [{'type' 'string'} {'type' 'null'}] 'title' 'Command' 'default' None 'de
scription' 'The command to use when starting a flow run In most cases this should be left blank and the command will be automatically generated by the worker ' } 'volumes' {'type' 'array' 'items' {'type' 'string'} 'title' 'Volumes' 'examples' ['/my/local/path /path/in/container'] 'description' 'A list of volume to mount into created containers '} 'networks' {'type' 'array' 'items' {'type' 'string'} 'title' 'Networks' 'description' 'Docker networks that created containers should be connected to '} 'mem_limit' { 'anyOf' [{'type' 'string'} {'type' 'null'}] 'title' 'Memory Limit' 'default' None 'description' ' ' } 'privileged'
k
@Marvin let's try again, I'm posting the output, will just remove a few more parts:
Copy code
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect version
prefect config view --hide-defaults --show-sources
Version:              3.4.21
API version:          0.8.4
Python version:       3.10.12
Git commit:           4fa9d9b4
Built:                Thu, Oct 02, 2025 07:14 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          server
Pydantic version:     2.11.9
Server:
  Database:           sqlite
  SQLite version:     3.37.2
Integrations:
  prefect-kubernetes: 0.6.5
  prefect-docker:     0.6.6
🚀 you are connected to:
<http://127.0.0.1:4200>
PREFECT_PROFILE='ephemeral'
PREFECT_API_URL='<http://127.0.0.1:4200/api>' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
(py310)  /workspace  ↱ 815-poc-workflow-orchestration ±  prefect work-pool ls -v
prefect work-pool inspect fraio
prefect work-queue ls -p fraio -v
prefect work-queue inspect default -p fraio
                                                                                                              Work Pools                                                                                                               
┏┳┳━━━━━━━━━━━━━━━━━━━┳┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃┃┃                ID ┃┃ Base Job Template                                                                                                                                                                                            ┃
┡╇╇━━━━━━━━━━━━━━━━━━━╇╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│││ 3a3d730e-86d9-49… ││ {'variables': {'type': 'object', '$defs': {'DockerRegistryCredentials': {'type': 'object', 'title': 'DockerRegistryCredentials', 'required': ['username', 'password', 'registry_url'], 'properties': {'reau… │
└┴┴───────────────────┴┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                      (**) denotes a paused pool                                                                                                       
WorkPool(
    id='3a3d730e-86d9-498b-a545-7880592d5b9f',
    created=DateTime(2025, 10, 4, 8, 3, 9, 206793, tzinfo=Timezone('UTC')),
    updated=DateTime(2025, 10, 4, 8, 4, 25, 245636, tzinfo=Timezone('UTC')),
    name='fraio',
    type='docker',
    base_job_template={
        'variables': {
            'type': 'object',
            '$defs': {
               
                }
            },
            'description': "..."
        },
        'job_configuration': {
            ...
        }
    },
    status=WorkPoolStatus.READY,
    storage_configuration=WorkPoolStorageConfiguration(),
    default_queue_id='834115d9-7a00-49d7-9c6a-49c1be0d2098'
)
                     Work Queues in Work Pool 'fraio'                      
┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name    ┃ Priority ┃ Concurrency Limit ┃ Description                    ┃
┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ default │ 1        │ None              │ The work pool's default queue. │
└─────────┴──────────┴───────────────────┴────────────────────────────────┘
                        (**) denotes a paused queue                        
WorkQueue(
    id='834115d9-7a00-49d7-9c6a-49c1be0d2098',
    created=DateTime(2025, 10, 4, 8, 3, 9, 210316, tzinfo=Timezone('UTC')),
    updated=DateTime(2025, 10, 4, 8, 18, 39, 659032, tzinfo=Timezone('UTC')),
    name='default',
    description="...",
    work_pool_name='fraio',
    work_pool_id=UUID('3a3d730e-86d9-498b-a545-7880592d5b9f'),
    last_polled=DateTime(2025, 10, 4, 8, 18, 39, 658388, tzinfo=Timezone('UTC')),
    status=WorkQueueStatus.READY
)
WorkQueueStatusDetail(healthy=False, late_runs_count=1, last_polled=DateTime(2025, 10, 4, 8, 18, 39, 658388, tzinfo=Timezone('UTC')), health_check_policy=WorkQueueHealthPolicy())
m
✅ thought for 71.5 seconds
Thanks for the outputs — they’re helpful. What they tell us - You’re on Prefect 3.4.21 and connected to http://127.0.0.1:4200, so CLI and UI are talking to the same API. - Pool fraio is READY and is a docker pool — good. - Queue default is READY and shows last_polled being updated recently — this means a worker is polling that queue. - Queue health shows late_runs_count=1 — there’s at least one run that reached its scheduled time but was not picked up. Given that the queue is being polled, the two most likely explanations are: 1) The deployment is not actually assigned to fraio/default (so the worker never sees those runs). 2) The worker tries to submit runs but hits a submission error (e.g., missing image/job variables, no Docker daemon access). Those errors would appear in the worker logs. Let’s verify the deployment and grab worker logs Please run and paste these: - Confirm deployment targeting:
Copy code
prefect deployment inspect "trigger-flow/fraio-trigger"
Look for work_pool_name and work_queue_name. They should be "fraio" and "default". - Bump worker logging and watch it while scheduling a run:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_WORKERS_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_CLIENT_LEVEL=DEBUG
prefect worker start --pool fraio --work-queue default
In another terminal, trigger a run and watch the worker output:
Copy code
prefect deployment run "trigger-flow/fraio-trigger"
If there’s a Docker submission problem (e.g., missing image, cannot connect to Docker), it should show up here. If you want to quickly isolate Docker from the equation - Create a process pool and try the same deployment against it:
Copy code
prefect work-pool create test-proc -t process --overwrite
prefect worker start --pool test-proc
Then redeploy or run the deployment against test-proc. If it runs there, the issue is Docker submission/networking. Notes for Docker networking (for later, once pickup works) Your PREFECT_API_URL is set to 127.0.0.1; containers started by the Docker worker won’t reach that. When you set up the deployment/job variables, point containers at an address reachable from inside Docker, e.g.: - Mac/Windows: http://host.docker.internal:4200/api - Linux: your host’s IP or expose the server on 0.0.0.0 and use that IP Docs: - Workers concept - Work queue CLI - Server and networking tips Once you share the deployment inspect and the worker DEBUG logs around the time you trigger a run, I can pinpoint the exact cause and provide a precise fix.
k
@Marvin let's go step by step:
Copy code
$ prefect deployment inspect "trigger-flow/fraio-trigger"
{
    'id': 'bdbbb4c5-1672-4fb0-84e1-8bc420d533f0',
    'created': '2025-10-04T08:05:43.700882+02:00',
    'updated': '2025-10-04T08:29:23.834869+02:00',
    'name': 'fraio-trigger',
    'version': 'local',
    'version_id': None,
    'version_info': None,
    'branch': None,
    'base': None,
    'root': None,
    'description': 'None',
    'flow_id': 'a62171d7-0b6e-432e-9eb6-b43aac22d657',
    'concurrency_limit': None,
    'global_concurrency_limit': {
        'id': '85d8ce36-1b01-44c6-b6cd-12374180c897',
        'created': '2025-10-04T08:05:43.720381+02:00',
        'updated': '2025-10-04T08:05:43.720390+02:00',
        'active': True,
        'name': 'deployment:bdbbb4c5-1672-4fb0-84e1-8bc420d533f0',
        'limit': 5,
        'active_slots': 0,
        'slot_decay_per_second': 0.0
    },
    'concurrency_options': None,
    'paused': False,
    'schedules': [],
    'job_variables': {'image': '<my-image-base>/fraio_algo:local'},
    'parameters': {},
    'pull_steps': [],
    'tags': [],
    'labels': {'prefect.flow.id': 'a62171d7-0b6e-432e-9eb6-b43aac22d657'},
    'work_queue_name': 'default',
    'last_polled': '2025-10-04T08:29:23.835089+02:00',
    'parameter_openapi_schema': {'type': 'object', 'title': 'Parameters', 'required': ['payload'], 'properties': {'payload': {'type': 'object', 'title': 'payload', 'position': 0, 'additionalProperties': True}}},
    'path': None,
    'entrypoint': 'fraio_orchestrator.fraio_trigger_flow.trigger_flow',
    'storage_document_id': None,
    'infrastructure_document_id': None,
    'created_by': None,
    'updated_by': None,
    'work_queue_id': None,
    'enforce_parameter_schema': True,
    'work_pool_name': 'fraio',
    'status': 'READY',
    'automations': []
m
✅ thought for 46.9 seconds
Great — thanks. Your deployment looks correctly targeted: - work_pool_name: fraio - work_queue_name: default - status: READY - image set in job_variables The suspicious bit is from your earlier queue details: - WorkQueueStatusDetail shows healthy=False and last_polled at 081839, while your deployment was updated around 08:29. That suggests your worker likely isn’t polling anymore (stopped/crashed), which would explain the Late run. Next step: confirm whether the worker is actually polling right now - Run this a couple of times 10–20s apart and see if last_polled advances:
Copy code
prefect work-queue ls -p fraio -v
If last_polled does not change, the worker is not polling. Run the worker in the foreground with debug logging and share its output
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_WORKERS_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_CLIENT_LEVEL=DEBUG

prefect worker start --pool fraio --work-queue default
Then, in another terminal, trigger a run and watch the worker:
Copy code
prefect deployment run "trigger-flow/fraio-trigger"
What to look for in the worker output - If the worker exits shortly after start or logs an error like: - “Cannot connect to the Docker daemon” or “Error while fetching server API version”: your Docker worker can’t reach the Docker daemon. Make sure the host’s Docker socket is accessible (e.g., /var/run/docker.sock mounted if you’re inside a container) or run the worker on the host where Docker is available. - Any auth/pull errors for your image: ensure the registry is reachable and credentials provided if needed. - If there are no errors and you see polling logs but still no claim, re-run the queue ls to see if last_polled updates. If it updates but runs remain Late, paste the worker logs around the time of the trigger. Side note for later (after pickup works): your flow container will not be able to reach 127.0.0.1:4200. When you move to running inside Docker, set PREFECT_API_URL for the flow container to something reachable from inside Docker, e.g. http://host.docker.internal:4200/api (Mac/Windows) or your host IP on Linux. Docs for reference: - Workers concept - Work queue CLI - Server and networking tips Once we see the worker’s DEBUG logs or confirm last_polled behavior, we can pinpoint the exact cause.
k
@Marvin
Copy code
10:32:52.443 | DEBUG   | prefect.events.clients - Pong received. Websocket connected.
10:32:52.443 | DEBUG   | prefect.events.clients - Resending 0 unconfirmed events.
10:32:52.444 | DEBUG   | prefect.events.clients - Finished resending unconfirmed events.
10:32:52.445 | DEBUG   | prefect.client - Connecting to API at <http://127.0.0.1:4200/api/>
10:32:52.445 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Emitting event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:32:52.446 | DEBUG   | prefect.events.clients - Added event id=0199ae5a-38aa-7535-a728-0a79a746363d to unconfirmed events list. There are now 1 unconfirmed events.
10:32:52.446 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Emit reconnection attempt 0.
10:32:52.446 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Sending event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:32:52.447 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Checkpointing event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:33:01.079 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:01.079 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:11.079613+02:00
10:33:01.091 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:10.033 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:10.033 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:20.033645+02:00
10:33:10.047 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:18.343 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:18.343 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:28.343831+02:00
10:33:18.352 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:25.943 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:25.943 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:35.943655+02:00
10:33:25.959 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:27.566 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend'
10:33:27.581 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server. 
10:33:36.709 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:36.709 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:46.709655+02:00
10:33:36.721 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:44.550 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:44.551 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:54.550989+02:00
10:33:44.562 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:57.167 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:57.167 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:34:07.167725+02:00
10:33:57.181 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:34:00.313 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend'
10:34:00.354 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server. 
10:34:06.296 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:34:06.296 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:34:16.296693+02:00
10:34:06.312 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:34:13.901 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:34:13.901 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:34:23.901596+02:00
10:34:13.914 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
So polling doesn't seem to find the flow run, although it does show in the prefect UI
m
Your message was too long, here's your message at the allowed limit: ```10 32 52 443 | DEBUG | prefect events clients - Pong received Websocket connected 10 32 52 443 | DEBUG | prefect events clients - Resending 0 unconfirmed events 10 32 52 444 | DEBUG | prefect events clients - Finished resending unconfirmed events 10 32 52 445 | DEBUG | prefect client - Connecting to API at <http //127 0 0 1 4200/api/> 10 32 52 445 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Emitting event id=0199ae5a-38aa-7535-a728-0a79a746363d 10 32 52 446 | DEBUG | prefect events clients - Added event id=0199ae5a-38aa-7535-a728-0a79a746363d to unconfirmed events list There are now 1 unconfirmed events 10 32 52 446 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Emit reconnection attempt 0 10 32 52 446 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Sending event id=0199ae5a-38aa-7535-a728-0a79a746363d 10 32 52 447 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Checkpointing event id=0199ae5a-38aa-7535-a728-0a79a746363d 10 33 01 079 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 01 079 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 11 079613+02 00 10 33 01 091 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 10 033 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 10 033 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 20 033645+02 00 10 33 10 047 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 18 343 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 18 343 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 28 343831+02 00 10 33 18 352 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 25 943 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 25 943 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 35 943655+02 00 10 33 25 959 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 27 566 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'sync_with_backend' 10 33 27 581 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server 10 33 36 709 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 36 709 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 46 709655+02 00 10 33 36 721 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 44 550 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 44 551 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a -
k
@Marvin again, it seems it's polling, but not finding the workflow:
Copy code
10:32:52.443 | DEBUG   | prefect.events.clients - Pong received. Websocket connected.
10:32:52.443 | DEBUG   | prefect.events.clients - Resending 0 unconfirmed events.
10:32:52.444 | DEBUG   | prefect.events.clients - Finished resending unconfirmed events.
10:32:52.445 | DEBUG   | prefect.client - Connecting to API at <http://127.0.0.1:4200/api/>
10:32:52.445 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Emitting event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:32:52.446 | DEBUG   | prefect.events.clients - Added event id=0199ae5a-38aa-7535-a728-0a79a746363d to unconfirmed events list. There are now 1 unconfirmed events.
10:32:52.446 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Emit reconnection attempt 0.
10:32:52.446 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Sending event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:32:52.447 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Checkpointing event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:33:01.079 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:01.079 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:11.079613+02:00
10:33:01.091 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:10.033 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:10.033 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:20.033645+02:00
10:33:10.047 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:18.343 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:18.343 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:28.343831+02:00
10:33:18.352 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:25.943 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:25.943 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:35.943655+02:00
10:33:25.959 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:27.566 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend'
10:33:27.581 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server. 
10:33:36.709 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:36.709 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:46.709655+02:00
10:33:36.721 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:44.550 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:44.551 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:54.550989+02:00
10:33:44.562 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:57.167 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:57.167 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:34:07.167725+02:00
10:33:57.181 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:34:00.313 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend'
10:34:00.354 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server. 
10:34:06.296 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:34:06.296 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:34:16.296693+02:00
10:34:06.312 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:34:13.901 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:34:13.901 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:34:23.901596+02:00
10:34:13.914 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
m
Your message was too long, here's your message at the allowed limit: again it seems it's polling but not finding the workflow ```10 32 52 443 | DEBUG | prefect events clients - Pong received Websocket connected 10 32 52 443 | DEBUG | prefect events clients - Resending 0 unconfirmed events 10 32 52 444 | DEBUG | prefect events clients - Finished resending unconfirmed events 10 32 52 445 | DEBUG | prefect client - Connecting to API at <http //127 0 0 1 4200/api/> 10 32 52 445 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Emitting event id=0199ae5a-38aa-7535-a728-0a79a746363d 10 32 52 446 | DEBUG | prefect events clients - Added event id=0199ae5a-38aa-7535-a728-0a79a746363d to unconfirmed events list There are now 1 unconfirmed events 10 32 52 446 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Emit reconnection attempt 0 10 32 52 446 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Sending event id=0199ae5a-38aa-7535-a728-0a79a746363d 10 32 52 447 | DEBUG | prefect events clients - EventsClient(id=134641341672432) Checkpointing event id=0199ae5a-38aa-7535-a728-0a79a746363d 10 33 01 079 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 01 079 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 11 079613+02 00 10 33 01 091 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 10 033 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 10 033 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 20 033645+02 00 10 33 10 047 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 18 343 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 18 343 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 28 343831+02 00 10 33 18 352 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 25 943 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 25 943 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 35 943655+02 00 10 33 25 959 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 27 566 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'sync_with_backend' 10 33 27 581 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server 10 33 36 709 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 36 709 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08 33 46 709655+02 00 10 33 36 721 | DEBUG | prefect workers docker dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs 10 33 44 550 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 33 44
k
@Marvin let's cut the output a bit 🙂
Copy code
10:32:52.443 | DEBUG   | prefect.events.clients - Pong received. Websocket connected.
10:32:52.443 | DEBUG   | prefect.events.clients - Resending 0 unconfirmed events.
10:32:52.444 | DEBUG   | prefect.events.clients - Finished resending unconfirmed events.
10:32:52.445 | DEBUG   | prefect.client - Connecting to API at <http://127.0.0.1:4200/api/>
10:32:52.445 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Emitting event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:32:52.446 | DEBUG   | prefect.events.clients - Added event id=0199ae5a-38aa-7535-a728-0a79a746363d to unconfirmed events list. There are now 1 unconfirmed events.
10:32:52.446 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Emit reconnection attempt 0.
10:32:52.446 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Sending event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:32:52.447 | DEBUG   | prefect.events.clients - EventsClient(id=134641341672432): Checkpointing event id=0199ae5a-38aa-7535-a728-0a79a746363d.
10:33:01.079 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:01.079 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:11.079613+02:00
10:33:01.091 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:10.033 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:10.033 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:20.033645+02:00
10:33:10.047 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:18.343 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:18.343 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:28.343831+02:00
10:33:18.352 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:25.943 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:25.943 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:35.943655+02:00
10:33:25.959 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
10:33:27.566 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend'
10:33:27.581 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Worker synchronized with the Prefect API server. 
10:33:36.709 | DEBUG   | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs'
10:33:36.709 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Querying for flow runs scheduled before 2025-10-04 08:33:46.709655+02:00
10:33:36.721 | DEBUG   | prefect.workers.docker.dockerworker f6c4e84b-987b-40ee-ab77-f522743fec3a - Discovered 0 scheduled_flow_runs
m
✅ thought for 115.4 seconds
k
@Marvin this may be related: also, I'd expect the db to be postgres, but it seems to be sqlite?
Copy code
prefect:
    container_name: prefect
    image: prefecthq/prefect:3.4.18-python3.12
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
    environment:
      - PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
      - PREFECT_API_URL=<http://127.0.0.1:4200/api>
      - PREFECT_SERVER_API_HOST=0.0.0.0
      - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://...
    ports:
      - 4200:4200
    depends_on:
      - postgres
m
✅ thought for 70.2 seconds
Thanks — those logs are clear: your Docker worker is polling fine but “Discovered 0 scheduled_flow_runs” each cycle. That means the API is not returning any runs for that queue/pool at those times, even though the UI shows a Late run. Let’s verify what the API assigned to the actual flow run and whether the run was really created with valid params. Please run these and paste the outputs: 1) Do you see any Scheduled/Late runs at all?
Copy code
prefect flow-run ls --limit 20 --state SCHEDULED --state LATE
2) Pick one Late run id from the list and inspect it. We want to see state, work_pool_id/work_queue_id, and parameters:
Copy code
prefect flow-run inspect <FLOW_RUN_ID>
Key things we’re checking for: - work_queue_id/work_pool_id are set on the flow run and match pool ‘fraio’ and queue ‘default’. - The run’s state is SCHEDULED (Late is okay as a derived state) and not something else that the worker won’t pick up. - The run has valid parameters — your deployment requires a parameter named ‘payload’ (enforce_parameter_schema=True). If a run was created without that parameter, creation may have failed or the UI may not have submitted it correctly. To create a known-good run right now, please trigger with the required parameter and watch it:
Copy code
prefect deployment run "trigger-flow/fraio-trigger" --param payload='{"test": 1}' --watch
Then, while the worker is running with DEBUG logging, you should see it get picked up. If the worker still shows “Discovered 0 scheduled_flow_runs” after you submit the run above, the flow-run inspect will tell us why (e.g., wrong queue assignment or invalid state). Also, run this a few seconds apart to confirm that the worker is actively polling and the queue’s last_polled is advancing:
Copy code
prefect work-queue ls -p fraio -v
If the flow run is correctly assigned and scheduled but still not discovered, we’ll dig into: - Any queue filters (include/exclude) that might be set; please share the full output of
Copy code
prefect work-queue inspect default -p fraio
- A potential mismatch in state (e.g., Pending vs Scheduled) on the flow run. Docs for reference: - Workers: Workers concept - Work queue CLI: Work queue CLI - Deployment run CLI: prefect deployment run Once we see the flow-run inspect for a Late run and the result of the param-validated run, we can pinpoint the exact issue.
Good catch — your server reporting “Database: sqlite” while your compose sets a Postgres URL is a red flag. That won’t directly cause runs to be Late, but it does indicate your server isn’t picking up the DB setting as intended. Let’s fix that first, then circle back to the worker. Why this can happen - The env var name is wrong or not applied in the container. - The DB URL contains special characters and needs quoting in YAML. - The server image/version mismatch or an early start before Postgres is ready. - You’re viewing config from a different server than the container you think you’re using. Recommended compose changes - Use a mapping for environment and quote the DB URL. - Match your image version to your CLI (3.4.21). - Don’t set PREFECT_API_URL inside the server container; it’s for clients. The server binds via PREFECT_SERVER_API_HOST/PORT. - Ensure Postgres is healthy before Prefect starts. Example
Copy code
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: prefect
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect -d prefect"]
      interval: 5s
      timeout: 3s
      retries: 10

  prefect:
    container_name: prefect
    image: prefecthq/prefect:3.4.21-python3.12
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
    environment:
      PREFECT_SERVER_API_HOST: "0.0.0.0"
      PREFECT_UI_URL: "<http://127.0.0.1:4200/ui>"
      # IMPORTANT: quote this URL in YAML
      PREFECT_API_DATABASE_CONNECTION_URL: "<postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>"
    ports:
      - "4200:4200"
    depends_on:
      postgres:
        condition: service_healthy

volumes:
  prefect:
Validate the server is actually using Postgres - After recreating the stack, check the server container’s view of settings:
Copy code
docker compose exec prefect prefect config view --hide-defaults --show-sources
- It should report Server: Database: postgresql and show PREFECT_API_DATABASE_CONNECTION_URL from env. - Also check logs for migrations:
Copy code
docker compose logs -f prefect
Important: if your DB URL has special characters (e.g., @, :, ?), keep it quoted as above. Back to the Late runs Once the server is confirmed healthy on Postgres, let’s verify the worker: 1) Confirm the queue is being polled now (run twice ~15s apart):
Copy code
prefect work-queue ls -p fraio -v
last_polled should advance. 2) Start the worker with DEBUG logs and trigger a run:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_WORKERS_LEVEL=DEBUG
export PREFECT_LOGGING_LOGGERS_PREFECT_CLIENT_LEVEL=DEBUG
prefect worker start --pool fraio --work-queue default
# in another terminal
prefect deployment run "trigger-flow/fraio-trigger"
If the worker shows Docker submission errors (cannot connect to daemon, image pull/auth issues), we’ll fix those next. If it polls but never claims, paste the worker logs around the trigger time. Docs - Prefect Server + Docker (networking/settings) - Workers concept - Work queue CLI Send back: - The result of config view inside the server container - The worker DEBUG logs when you trigger a run That’ll let me pinpoint the remaining cause for the Late runs.
k
@Marvin afte some more debugging, it got it solved, using python modules for deployment. thx.
m
✅ thought for 13.6 seconds
Awesome — glad you got it working! Using Python modules for deployment is a great move with Prefect 3.x; it keeps deployments versioned and reproducible. Quick tips to keep things smooth: - Prefer Python-based deployments (
Flow.deploy
or 
flow.from_source(...).deploy(...)
) over CLI build commands in 3.x. - Keep the server, CLI, and worker on matching minor versions when possible. - For Docker runs, ensure the flow container can reach the API: - Mac/Windows: set 
PREFECT_API_URL
to 
<http://host.docker.internal:4200/api>
- Linux: use your host IP or bind the server to 
0.0.0.0
and use that IP - If you move to Postgres, quote your DB URL in compose and confirm the server picks it up via 
prefect config view
inside the container. Docs you might find handy: - Deployments in 3.x: Deploy with Docker - Workers concept: Workers overview - Self-hosted server + Docker: Server/Docker setup If you want, I can share a minimal Python deployment snippet tailored to your pool/image setup or help validate your Docker/DB config.
4 Views
Open in Slack
PreviousNext
Powered by