https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> let's try once more: i've got a very...
# marvin-ai
j
@Marvin let's try once more: i've got a very simple setup via docker compose that looks like this:
Copy code
# Docker Compose file that defines the Prefect Server and its associated workpools

version: '3.8'
services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    container_name: prefect-server
    ports:
      - "8080:4200"
    env_file:
      - /home/prefect/config.env
        #command: prefect server start --host 0.0.0.0
    command: /bin/bash -c "pip install prefect-gcp prefect-redis && prefect server start --host 0.0.0.0"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - prefect-network
    restart: always
networks:
  prefect-network:
    driver: bridge
with config.env that looks like this:
Copy code
PREFECT_HOME=/home/prefect
PREFECT_SERVER_API_BASE_PATH=/prefect/api
PREFECT_SERVER_UI_SERVE_BASE=/prefect
PREFECT_API_URL=<http://10.142.0.18:8080/prefect/api>
PREFECT_SQLALCHEMY_POOL_SIZE=500
PREFECT_SQLALCHEMY_MAX_OVERFLOW=500


PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_EXTRA_LOGGERS=prefect.orchestration.services,prefect.client,prefect.flow_runner,prefect.task_runner,sqlalchemy,asyncpg,urllib3
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_REDIS_MESSAGING_DB=0
PREFECT_REDIS_MESSAGING_HOST=10.142.0.16
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=120
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=180
PREFECT_SERVER_DATABASE_TIMEOUT=60
when i run tasks, everything works as expected except one thing: the tasks do not show up in the UI or in the "task tabs". the flow completes and everything is populated as it shoudl be, but there's a constant "this flow generated no sub-tasks" messages. i can access redis, the API, and every relevant URL from within the docker container so this is not an accessibility issue. do you have anny idea what's going on ?
m
✅ thought for 44.0 seconds
Based on the research and your setup, I've identified a few potential issues and solutions: 1. API URL Configuration Issue Your 
PREFECT_API_URL
is using 
<http://10.142.0.18:8080/prefect/api>
, but you're mapping port 8080 to 4200 in your Docker Compose file. This might cause a mismatch. The container is listening on port 4200 internally, but you're accessing it on 8080 externally. Make sure these align. 2. Missing Worker Configuration I notice your compose file only includes the server, but not any workers. In Prefect 3.x, you need workers running to execute your flows. Are you running workers separately? If not, you'll need to set up and run workers to properly execute and display tasks. 3. Redis Configuration Your Redis configuration looks correct, but a few suggestions: - Consider adding 
PREFECT_REDIS_MESSAGING_PASSWORD
if your Redis instance requires authentication - You might want to add 
PREFECT_REDIS_MESSAGING_TIMEOUT=180
to match your other timeout settings Here are some steps to troubleshoot: 1. First, verify your API connectivity from within the container:
Copy code
bash
docker exec prefect-server curl <http://localhost:4200/prefect/api/health>
2. Check the Prefect server logs for any relevant errors:
Copy code
bash
docker logs prefect-server
3. Verify your flow code is using tasks correctly. In Prefect 3.x, tasks should be defined using the 
@task
decorator:
Copy code
python
from prefect import flow, task

@task
def my_task():
    return "Hello"

@flow
def my_flow():
    result = my_task()
4. Consider modifying your Docker Compose file to use the internal port 4200 consistently:
Copy code
yaml
ports:
  - "4200:4200"
And then update your 
PREFECT_API_URL
to:
Copy code
PREFECT_API_URL=<http://10.142.0.18:4200/prefect/api>
Could you: 1. Confirm if you have workers running and properly configured? 2. Share a simple example of one of your flow/task definitions that's not showing up in the UI? 3. Check if you see any relevant errors in the server logs? This will help us narrow down the exact cause of the issue.
j
@Marvin dude everything is properly configured and defined. when i remove the REDIS configurations everythign shows up as expected. the only thing at fault here regards the redis configuration --- and the container is able to reach and contact the redis instance (i can 
redis-cli
etc.)
m
✅ thought for 30.2 seconds
Looking at your configuration, I notice you're using both:
Copy code
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
Try setting only the broker, and use the default cache implementation:
Copy code
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
# Remove PREFECT_MESSAGING_CACHE=prefect_redis.messaging
If that doesn't work, could you try adding these debug settings temporarily to see if we can get more information about what's happening with the Redis messaging:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_EXTRA_LOGGERS=prefect_redis.messaging
Also, could you verify the Redis connection by running this command inside your container:
Copy code
bash
python3 -c "from prefect_redis.messaging import RedisMessagingBlock; RedisMessagingBlock(host='10.142.0.16', port=6379, database=0).test_connection()"
This is a somewhat unusual issue - if Redis is reachable but tasks aren't showing up only when Redis messaging is enabled, it suggests there might be an issue with how the task state messages are being handled by the Redis messaging backend. The fact that everything works without Redis points to this being a configuration issue rather than a connectivity one.
Open in Slack
PreviousNext
Powered by