Hello! I've encountered a Prefect-related issue so I am looking for some help from the community. First let me say that I am new to Prefect. I have a dockerized Python script that's supposed to run on the same Docker network via Docker Compose with Postgres and Prefect containers. The Docker Compose YAML file looks like this:

services:
  pgdatabase:
    image: postgres:13
    environment:
      - POSTGRES_USER=*****
      - POSTGRES_PASSWORD=*****
      - POSTGRES_DB=*****
    volumes:
      - "./ny_taxi_postgres_data:/var/lib/postgresql/data:rw"
    ports:
      - "5432:5432"
  pgadmin:
    image: dpage/pgadmin4
    environment:
      - PGADMIN_DEFAULT_EMAIL=*****
      - PGADMIN_DEFAULT_PASSWORD=*****
    ports:
      - "8080:80"
  prefect:
    image: prefecthq/prefect
    ports:
      - "4201:4200"
    environment:
      - PREFECT_API_URL=<http://localhost:4200/api>
      - PREFECT_API_REQUEST_TIMEOUT=60
      - PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=False

When I do:

docker run -it --network=my_docker_network   my_dockerized_script_image

I am getting this error:

16:58:15.968 | INFO    | prefect - Starting server on <http://127.0.0.1:8101>
Traceback (most recent call last):
  File "/app/ingest_data_parquet_prefect.py", line 99, in <module>
    main(args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 1345, in __call__
    return run_flow(
  File "/usr/local/lib/python3.10/site-packages/prefect/flow_engine.py", line 821, in run_flow
    return run_flow_sync(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/flow_engine.py", line 696, in run_flow_sync
    with engine.start():
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/site-packages/prefect/flow_engine.py", line 635, in start
    with self.initialize_run():
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/site-packages/prefect/flow_engine.py", line 554, in initialize_run
    with SyncClientContext.get_or_create() as client_ctx:
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/site-packages/prefect/context.py", line 231, in get_or_create
    with SyncClientContext() as ctx:
  File "/usr/local/lib/python3.10/site-packages/prefect/context.py", line 204, in __init__
    client=get_client(sync_client=True, httpx_settings=httpx_settings),
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 219, in get_client
    server.start()
  File "/usr/local/lib/python3.10/site-packages/prefect/server/api/server.py", line 817, in start
    raise RuntimeError(error_message)
RuntimeError: Timed out while attempting to connect to ephemeral Prefect API server.
16:58:26.244 | INFO    | prefect - Stopping server on <http://127.0.0.1:8101>

So it's a timeout error related to the ephemeral Prefect API Server. As you can see in the Docker Compose YAML file, I tried to set

PREFECT_API_REQUEST_TIMEOUT=60

and

PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=False

but no luck. I saw an older comment about this error but there wasn't really a solution posted, so I figured I should ask too. Has anyone seen this before, and if so, what's the solution? TIA!

I have something very similar to the following. I think the key environment variables that got it working were

PREFECT_SERVER_API_HOST

and

PREFECT_SERVER_API_PORT

. It also looks like you’re missing command

services:
  prefect:
    image: prefecthq/prefect:3.0-python3.12
    ports:
      - '4200:4200'
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: ${PREFECT_API_DATABASE_CONNECTION_URL}
      PREFECT_SERVER_API_HOST: "0.0.0.0"
      PREFECT_SERVER_API_PORT: 4200
    networks:
      - prefectnet
    volumes:
      - prefect:/root/.prefect
    command: [ "prefect", "server", "start" ]

  prefect-worker:
    image: prefecthq/prefect:3.0-python3.12
    environment:
      PREFECT_API_URL: <http://prefect:4200/api>
      EXTRA_PIP_PACKAGES: prefect-docker
    command: [ "prefect", "worker", "start", "--pool", "docker" ]
    networks:
      - prefectnet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  prefect:
    driver: local

networks:
  prefectnet:
    external: true

@Chris Reid thanks for sharing. I appreciate it. No luck though. What I've noticed is that when I run the image, the logs start with this:

20:16:23.335 | INFO    | prefect - Starting server on <http://127.0.0.1:8240>

On an identical machine that I had, for the same exact docker image, logs started with this:

/usr/local/lib/python3.10/contextlib.py:142: SAWarning: Skipped unsupported reflection of expression-based index ix_flow_run__coalesce_start_time_expected_start_time_desc
  next(self.gen)
/usr/local/lib/python3.10/contextlib.py:142: SAWarning: Skipped unsupported reflection of expression-based index ix_flow_run__coalesce_start_time_expected_start_time_asc
  next(self.gen)
02:49:33.894 | INFO    | prefect.engine - Created flow run 'piquant-labradoodle' for flow 'Ingest flow'

So it didn't have to start the prefect server, which is strange. I think the issue relates to this.