David Martin Calalang
08/04/2025, 2:35 PMMarvin
08/04/2025, 2:36 PMMarvin
08/04/2025, 2:36 PMPREFECT_API_URL
- This should point to your AWS ECS load balancer URL, for example:
bash
export PREFECT_API_URL="<https://your-load-balancer-url.amazonaws.com>"
2. Verifying/Setting the Configuration:
You can:
- Set it temporarily using the export command above
- Set it permanently in your Prefect profile using:
bash
prefect config set PREFECT_API_URL="<https://your-load-balancer-url.amazonaws.com>"
- View your current settings using:
bash
prefect config view
3. Flow Run Location:
- The PREFECT_API_URL
only controls where flows are registered and where their state is tracked
- The actual execution location of flow runs depends on your worker and work pool configuration
- When you run flows locally with flow.serve()
or python flow.py
, they will run on your local machine but report their status to the remote server
- For automated deployments, you'll need to set up workers in your execution environment (like ECS) that pull work from your work pools
4. Authentication (if needed):
- If your server requires authentication, you'll also need to set PREFECT_API_KEY
- This can be done similarly:
bash
export PREFECT_API_KEY="your-api-key"
# or
prefect config set PREFECT_API_KEY="your-api-key"
To verify your configuration is working:
1. Set the PREFECT_API_URL
as described above
2. Run prefect config view
to confirm the settings
3. Try running a simple flow - if it can communicate with your server, you'll see the flow run appear in the UI at your ECS-hosted server
Remember that this only configures where your client communicates with the Prefect server. The actual execution environment for your flows will depend on your deployment configuration and worker setup.David Martin Calalang
08/04/2025, 3:05 PMMarvin
08/04/2025, 3:05 PMMarvin
08/04/2025, 3:06 PMbash
export PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://username:password@your-rds-endpoint:5432/database_name>"
Or add it to your Prefect profile:
bash
prefect config set PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://username:password@your-rds-endpoint:5432/database_name>"
2. Connection Pool Settings:
For production environments, you should configure the connection pool:
bash
# Maximum number of database connections in the pool
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
# Connection recycling time in seconds
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
# Connection timeout in seconds
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60
3. PostgreSQL Notification Settings:
These control how Prefect handles PostgreSQL NOTIFY/LISTEN connections:
bash
# Interval for reconnecting to Postgres for notifications
export PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_RECONNECT_INTERVAL_SECONDS=60
# Heartbeat interval for Postgres notifications
export PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_HEARTBEAT_INTERVAL_SECONDS=30
4. Database Initialization:
Before starting the server, you need to initialize the database:
bash
prefect server database reset
⚠️ Warning: This command will reset your database. For production, you might want to just run migrations instead:
bash
prefect server database upgrade
5. Security Considerations:
- Ensure your RDS security group allows incoming connections from your Prefect server
- Use AWS Secrets Manager or environment variables to manage database credentials securely
- Consider using SSL for database connections by adding ?sslmode=require
to your connection URL
6. Example Complete Configuration:
Here's a complete example using an AWS RDS instance:
bash
# Database Connection
export PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://username:password@your-rds.xxxxx.region.rds.amazonaws.com:5432/prefect?sslmode=require>"
# Connection Pool Settings
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60
# Notification Settings
export PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_RECONNECT_INTERVAL_SECONDS=60
export PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_HEARTBEAT_INTERVAL_SECONDS=30
7. Verifying the Configuration:
After setting up, you can:
- Check your configuration: prefect config view
- Test database connectivity: prefect server database status
- Start the server: prefect server start
Remember to:
- Back up your database regularly
- Monitor connection pool usage and adjust pool settings based on your workload
- Keep track of database migrations when upgrading Prefect
- Consider using AWS IAM authentication for RDS if appropriate for your security requirements
If you encounter any connection issues, verify:
1. The RDS instance is accessible from your Prefect server's network
2. The security group rules allow the connection
3. The database user has appropriate permissions
4. The database exists and is properly initialized with Prefect schemas