Javier King
07/16/2025, 9:36 PMservices:
prefect-server:
image: prefecthq/prefect:3-latest
container_name: prefect-server
ports:
- "8080:4200"
env_file:
- /home/prefect/config.env
command: prefect server start --host 0.0.0.0 --no-services
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- prefect-network
restart: always
prefect-background-services:
image: prefecthq/prefect:3-latest
depends_on: [prefect-server]
container_name: prefect-background-services
command: /bin/bash -c "pip install prefect-gcp prefect-redis && prefect server services start"
environment:
- PREFECT_API_URL=<http://10.142.0.18:8080/prefect/api>
env_file:
- /home/prefect/config.env
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- prefect-network
restart: always
note: the API_URL provided here is the URL which forwards traffic to the machine on which these containers are run. these URLs, redis, and postgres are all accessible from both containers so its not an accessibility issueMarvin
07/16/2025, 9:36 PMMarvin
07/16/2025, 9:38 PM/home/prefect/config.env
- Ensure both PREFECT_API_URL
and messenger (Redis) settings are correctly configured
2. API URL Configuration
The current setup uses:
yaml
PREFECT_API_URL=<http://10.142.0.18:8080/prefect/api>
Try these troubleshooting steps:
- Verify this URL is accessible from within the services container:
bash
curl <http://10.142.0.18:8080/prefect/api/health>
- Consider using the container name instead:
yaml
PREFECT_API_URL=<http://prefect-server:4200/api>
3. Redis Configuration
- Ensure both containers have identical Redis configuration
- Verify Redis connection from both containers
- Check Redis logs for any connection issues
4. Debugging Steps
- Enable debug logging by adding to your config.env:
PREFECT_LOGGING_LEVEL=DEBUG
- Check logs in both containers for any connection issues
- Verify task state changes are being sent by checking Redis pub/sub messages
5. Network Configuration
Your network setup looks correct with both containers on prefect-network
, but verify:
- Both containers can resolve each other's hostnames
- No firewall rules are blocking container-to-container communication
### Recommendations
1. First, verify API connectivity:
bash
# From inside the services container
curl <http://10.142.0.18:8080/prefect/api/health>
2. Check service logs with increased verbosity:
bash
docker logs prefect-background-services
docker logs prefect-server
3. Verify Redis connectivity from both containers:
python
from prefect.client.schemas.actions import ClientAction
from prefect.client import get_client
# This will attempt to connect to both API and Redis
async with get_client() as client:
print("Connected successfully")
Would you like me to provide more specific debugging steps for any of these areas? Also, could you share:
1. The contents of your config.env file (excluding sensitive information)?
2. Are you seeing any specific behavior in the UI (like delayed updates, or completely missing tasks)?Javier King
07/16/2025, 9:40 PMPREFECT_HOME=/home/prefect
PREFECT_SERVER_API_BASE_PATH=/prefect/api
PREFECT_SERVER_UI_SERVE_BASE=/prefect
PREFECT_API_URL=/prefect/api
PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_EXTRA_LOGGERS=sqlalchemy,asyncpg,urllib3
PREFECT_SQLALCHEMY_POOL_SIZE=500
PREFECT_SQLALCHEMY_MAX_OVERFLOW=500
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_REDIS_MESSAGING_DB=0
PREFECT_REDIS_MESSAGING_HOST=10.142.0.16
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=120
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=180
PREFECT_SERVER_DATABASE_TIMEOUT=60
PREFECT_SQLALCHEMY_MAX_OVERFLOW=100
PREFECT_SQLALCHEMY_POOL_SIZE=100
i removed sensitive values. like i said, it is not an accessibility issue; all the URLs and connections work as expected from within the containers. the UI just doesnt show tasks on the chart, but logs are populated and the flow completes successfullyMarvin
07/16/2025, 9:41 PMMarvin
07/16/2025, 9:41 PMPREFECT_API_URL=/prefect/api
This is a relative URL, which won't work correctly for the services container. The services container needs an absolute URL to connect to the API server. You already have the correct URL in your docker-compose file, but it needs to be in the config.env file instead:
PREFECT_API_URL=<http://10.142.0.18:8080/prefect/api>
2. Potential Secondary Issue - Duplicate Configuration
You have both:
PREFECT_SQLALCHEMY_MAX_OVERFLOW=500
PREFECT_SQLALCHEMY_MAX_OVERFLOW=100
The second value would override the first, but this isn't related to your current issue.
### The Fix
1. Update your config.env to use the absolute URL:
PREFECT_API_URL=<http://10.142.0.18:8080/prefect/api>
2. Remove the PREFECT_API_URL from the docker-compose environment section since it will be in the config.env:
yaml
prefect-background-services:
image: prefecthq/prefect:3-latest
depends_on: [prefect-server]
container_name: prefect-background-services
command: /bin/bash -c "pip install prefect-gcp prefect-redis && prefect server services start"
env_file:
- /home/prefect/config.env
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- prefect-network
restart: always
### Why This Should Fix It
The services container needs to know exactly where to find the API server. When you provide a relative URL (/prefect/api
), the services don't know what base URL to use. By providing the absolute URL, the services can properly communicate state changes to the API server, which is necessary for the UI to display task information.
The reason your flows still complete successfully is that the actual flow execution and Redis messaging are working correctly - it's just the API communication for UI updates that's affected by the relative URL.
Try making these changes and let me know if you start seeing your tasks appear in the UI. If not, we can enable more detailed debugging by adding some additional loggers to your configuration.Javier King
07/16/2025, 9:44 PMdocker exec -it ... /bin/bash
into the container and looking at the value of PREFECT_API_URL, so it's not thatMarvin
07/16/2025, 9:44 PMMarvin
07/16/2025, 9:46 PMpython
from prefect.settings import PREFECT_REDIS_MESSAGING_HOST, PREFECT_REDIS_MESSAGING_PORT, PREFECT_REDIS_MESSAGING_DB
print(f"Redis Host: {PREFECT_REDIS_MESSAGING_HOST.value()}")
print(f"Redis Port: {PREFECT_REDIS_MESSAGING_PORT.value()}")
print(f"Redis DB: {PREFECT_REDIS_MESSAGING_DB.value()}")
2. Check if task states are being published to Redis
You can monitor Redis messages in real-time. From inside either container:
python
import redis
import json
r = redis.Redis(
host='10.142.0.16',
port=6379,
db=0
)
pubsub = r.pubsub()
pubsub.subscribe('prefect:*') # Subscribe to all Prefect channels
for message in pubsub.listen():
if message['type'] == 'message':
print(json.loads(message['data']))
3. Check the services that are actually running
In the services container:
bash
ps aux | grep prefect
You should see multiple service processes running.
4. Verify the orchestration services are receiving messages
Add these specific loggers to your config:
PREFECT_EXTRA_LOGGERS=prefect.orchestration.services,prefect.client,prefect.flow_runner,prefect.task_runner
Questions:
1. When you look at the UI, do you see the flow runs themselves? Is it just the task information that's missing?
2. In the task runs page of a flow run, do you see any task runs listed at all, or are they completely absent?
3. Are you using any custom task runners or flow runners?
4. Can you check if there are any messages in Redis at all related to Prefect? This command will show all keys:
bash
redis-cli -h 10.142.0.16 -p 6379 keys "prefect:*"
The fact that logs are populated but tasks aren't showing up suggests that the task state updates are either:
1. Not being published to Redis correctly
2. Not being consumed by the orchestration services
3. Not being properly processed by the API server
Let's start with checking these aspects and then we can narrow down where the communication chain is breaking.Javier King
07/16/2025, 10:12 PMMarvin
07/16/2025, 10:12 PMMarvin
07/16/2025, 10:14 PMPREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
But the services container needs to have the prefect-redis
package properly installed and available when the services start. In your docker-compose, you're installing it, but there might be a race condition where the services start before the installation completes:
yaml
command: /bin/bash -c "pip install prefect-gcp prefect-redis && prefect server services start"
Let's try a more robust approach:
1. First, create a new Dockerfile for your services:
dockerfile
FROM prefecthq/prefect:3-latest
RUN pip install prefect-gcp prefect-redis
CMD ["prefect", "server", "services", "start"]
2. Update your docker-compose.yml to use this custom image:
yaml
prefect-background-services:
build: . # assuming Dockerfile is in the same directory
depends_on: [prefect-server]
container_name: prefect-background-services
environment:
- PREFECT_API_URL=<http://10.142.0.18:8080/prefect/api>
env_file:
- /home/prefect/config.env
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- prefect-network
restart: always
3. Also, let's verify the Redis connection by adding this to your config.env:
PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_EXTRA_LOGGERS=prefect_redis.messaging,prefect.server.services,prefect.server.utilities.messaging
4. After making these changes, rebuild and restart your containers:
bash
docker-compose down
docker-compose build
docker-compose up -d
5. Then check the logs:
bash
docker logs prefect-background-services
The key things we're looking for in the logs are:
- Successful initialization of the Redis messaging broker
- Any errors related to Redis connections or messaging
- Service startup messages indicating which services are running
Can you try these changes and let me know what you see in the logs? Particularly, we want to confirm that:
1. The Redis messaging package is properly installed
2. The messaging broker is successfully initialized
3. There are no errors related to Redis connections
If you still don't see any Redis messages after these changes, we might need to verify the Redis messaging implementation itself is working as expected.Javier King
07/16/2025, 10:22 PMMarvin
07/16/2025, 10:22 PMMarvin
07/16/2025, 10:23 PMpython
from prefect import flow, task
@task
def my_task():
return "hello"
@flow
def my_flow():
result = my_task()
The fact that logs are populated but tasks aren't showing up is particularly interesting because it suggests that the task execution information is making it to the API server (hence the logs) but something might be wrong with how the task states are being processed or stored.
Also, could you check the API server logs specifically around the time a flow runs? Look for any messages containing "task" or "state":
bash
docker logs prefect-server | grep -i "task\|state"
This will help us understand if the API server is receiving the task state updates but perhaps not processing them correctly.Javier King
07/16/2025, 10:35 PMMarvin
07/16/2025, 10:35 PMMarvin
07/16/2025, 10:35 PMpython
import redis
r = redis.Redis(
host='10.142.0.16',
port=6379,
db=0
)
# List all streams
streams = r.keys('prefect:*')
print("Existing streams:", streams)
# For each stream, get the last few messages
for stream in streams:
if b'stream' in stream:
print(f"\nMessages in {stream.decode()}:")
messages = r.xread({stream: '0-0'}, count=5)
print(messages)
2. Let's also check your Redis configuration more thoroughly. What's the output of:
bash
docker exec -it prefect-background-services env | grep REDIS
3. Could you also try temporarily switching back to the default memory messaging to confirm it's specifically a Redis issue?
bash
# Remove these from your config.env temporarily
# PREFECT_MESSAGING_BROKER=prefect_redis.messaging
# PREFECT_MESSAGING_CACHE=prefect_redis.messaging
Based on the issue I found, there might be a problem with how task states are being persisted when using Redis messaging. The states are being published but might not be getting properly consumed and persisted to the database.
Let me know the results of these tests and we can determine the best path forward. There might be a few potential solutions:
1. Upgrade to the latest version of prefect-redis if you're not already on it
2. Add some retry logic for state persistence
3. Configure Redis stream settings differently
4. Use a workaround with the default memory messaging if needed
Which would you like to try first?Javier King
07/16/2025, 10:42 PM"message:messaging-cache:cd5b645f-66a6-47f4-ac4a-03b5ecd9083c"
being in Redis mean anything to you?
also can you give me some prefect commands that diagnose the redis cache? something like prefect cache view
Marvin
07/16/2025, 10:42 PMMarvin
07/16/2025, 10:43 PMmessage:messaging-cache:[uuid]
pattern indicates that Prefect is using Redis for message caching as part of its messaging system. This is part of Prefect's distributed messaging infrastructure.
While there isn't a direct prefect cache
command, you can inspect the Redis cache directly using Redis CLI commands. Here's how to diagnose the Redis cache:
1. Connect to Redis and inspect the key:
bash
# From inside the container
redis-cli -h 10.142.0.16 -p 6379
# Once connected:
TYPE message:messaging-cache:cd5b645f-66a6-47f4-ac4a-03b5ecd9083c
GET message:messaging-cache:cd5b645f-66a6-47f4-ac4a-03b5ecd9083c
TTL message:messaging-cache:cd5b645f-66a6-47f4-ac4a-03b5ecd9083c
2. List all Prefect-related keys:
bash
# Show all Prefect messaging cache keys
KEYS message:messaging-cache:*
# Show all Prefect-related keys
KEYS prefect:*
3. Monitor Redis in real-time while a flow runs:
bash
MONITOR
The presence of that key suggests that the Redis messaging system is working at some level - messages are being cached. However, if task states aren't showing up in the UI, there might be an issue with:
1. Message consumption
2. Message persistence
3. Cache expiration
Could you try running these Redis commands and share:
1. What type of value is stored in that cache key?
2. Does it have a TTL set?
3. Do you see new cache keys being created when you run flows?
This will help us understand if the messaging system is functioning correctly but perhaps not being consumed properly, or if messages are expiring too quickly.
Also, you might want to check the Redis stream keys specifically:
bash
# In redis-cli
KEYS prefect:stream:*
If you see stream keys but no messages are being processed, we might need to look at the consumer groups:
bash
# Replace [stream-key] with an actual stream key you find
XINFO GROUPS [stream-key]
This will show us if the consumer groups are properly set up and processing messages.