<@ULVA73B9P> - where does flow code execute? Does ...
# ask-marvin
j
@Marvin - where does flow code execute? Does any flow code actually execute on the prefect server vs the worker? There is concern around data being passed between server and worker
m
thought for 222.6 seconds
Short answer: your flow code never runs on the Prefect server. It runs where your worker (or
flow.serve()
process) runs. The server just orchestrates and stores metadata. Where code executes - Worker-based runs: A worker polls your work pool, pulls scheduled flow runs, and executes your flow in the configured infrastructure (local process, Docker, Kubernetes, etc.). No user code is executed by the server. - Serve/local runs: If you use
flow.serve()
, that long-lived process polls for runs and executes your flow locally. Again, not on the server. What data moves between worker and server - From server to worker: - Flow run metadata: run ID, state, deployment info, job variables - Parameters: serialized as JSON and included on the flow run - From worker to server: - State transitions: e.g., Pending → Running → Completed/Failed (with timestamps and messages) - Logs: batched to the API if enabled - Heartbeats: lightweight worker health/metadata - Result references: if you persist results, the API stores a reference (metadata) to the object in your configured result storage, not the data itself Where results live - By default, results are written to a result storage location (local filesystem or a storage block like S3/Azure/GCS). The API stores a small reference to that object so retries/caching can fetch it. - Large data is not sent to the server; only the result reference is. - You can point result storage to your own infrastructure to keep data entirely within your network. Controlling what’s shared - Parameters: Anything you pass as parameters is stored on the server in the flow run record (JSON). Avoid putting secrets or sensitive data in parameters. - Logs: You can disable log shipping to the API with
PREFECT_LOGGING_TO_API_ENABLED=false
, or keep logs but avoid logging sensitive content. - Results: Configure result storage to an on-prem or private bucket. You can also adjust persistence behavior per flow/task or via settings if you want to avoid persisting results for certain runs. Helpful docs - Results and result storage: Results - Logging to the API: Logs - Workers and work pools: Workers &amp; Work Pools - Deployments overview: Deployments If you can share a bit more about your environment (Prefect Cloud vs. self-hosted, and what specific data you’re worried about), I can suggest concrete settings and patterns to keep everything within your security boundaries.