Joseph Wibowo
11/19/2025, 10:05 PMMarvin
11/19/2025, 10:05 PMMarvin
11/19/2025, 10:09 PMflow.serve() process) runs. The server just orchestrates and stores metadata.
Where code executes
- Worker-based runs: A worker polls your work pool, pulls scheduled flow runs, and executes your flow in the configured infrastructure (local process, Docker, Kubernetes, etc.). No user code is executed by the server.
- Serve/local runs: If you use flow.serve(), that long-lived process polls for runs and executes your flow locally. Again, not on the server.
What data moves between worker and server
- From server to worker:
- Flow run metadata: run ID, state, deployment info, job variables
- Parameters: serialized as JSON and included on the flow run
- From worker to server:
- State transitions: e.g., Pending → Running → Completed/Failed (with timestamps and messages)
- Logs: batched to the API if enabled
- Heartbeats: lightweight worker health/metadata
- Result references: if you persist results, the API stores a reference (metadata) to the object in your configured result storage, not the data itself
Where results live
- By default, results are written to a result storage location (local filesystem or a storage block like S3/Azure/GCS). The API stores a small reference to that object so retries/caching can fetch it.
- Large data is not sent to the server; only the result reference is.
- You can point result storage to your own infrastructure to keep data entirely within your network.
Controlling what’s shared
- Parameters: Anything you pass as parameters is stored on the server in the flow run record (JSON). Avoid putting secrets or sensitive data in parameters.
- Logs: You can disable log shipping to the API with PREFECT_LOGGING_TO_API_ENABLED=false, or keep logs but avoid logging sensitive content.
- Results: Configure result storage to an on-prem or private bucket. You can also adjust persistence behavior per flow/task or via settings if you want to avoid persisting results for certain runs.
Helpful docs
- Results and result storage: Results
- Logging to the API: Logs
- Workers and work pools: Workers & Work Pools
- Deployments overview: Deployments
If you can share a bit more about your environment (Prefect Cloud vs. self-hosted, and what specific data you’re worried about), I can suggest concrete settings and patterns to keep everything within your security boundaries.