https://prefect.io
Join Slack
<@ULVA73B9P> how to run deployment remotely and get the return output back in response
# ask-marvin
s
@Marvin how to run deployment remotely and get the return output back in response
m
✅ thought for 104.4 seconds
In Prefect 3.x, the simplest way is to trigger the deployment and wait for the flow run to finish, then read the result from the returned state. Recommended approach: 
run_deployment()
Copy code
from prefect.deployments import run_deployment

flow_run = run_deployment(
    name="my-flow/my-deployment",
    parameters={"x": 1},
    timeout=None,   # wait until finished
)

result = flow_run.state.result()
print(result)
If the deployment completes successfully, 
result
is the flow's return value. Async version:
Copy code
from prefect.deployments import arun_deployment

flow_run = await arun_deployment(
    name="my-flow/my-deployment",
    parameters={"x": 1},
    timeout=None,
)

result = await flow_run.state.aresult()
print(result)
Important limitation If you trigger a deployment through the raw REST API or lower-level client, the initial response only gives you the flow run metadata — not the final return value. You must: 1. create the flow run 2. poll until it reaches a final state 3. fetch the result from the state Example with the client:
Copy code
import asyncio
from prefect import get_client

async def run_and_get_result(deployment_id):
    async with get_client() as client:
        flow_run = await client.create_flow_run_from_deployment(
            deployment_id=deployment_id,
            parameters={"x": 1},
        )

        while True:
            flow_run = await client.read_flow_run(flow_run.id)
            if flow_run.state.is_final():
                break
            await asyncio.sleep(2)

        return await flow_run.state.aresult()
A few gotchas: - The flow’s return value must be serializable / storable as a Prefect result. - If the flow fails, 
state.result()
/ 
state.aresult()
can raise the underlying exception. - 
timeout=0
returns immediately, so you won’t have the final output yet. If you want, I can give you: - a FastAPI example that exposes “run deployment and return result” - a curl/REST polling example - or a pattern for large outputs (store artifact/object and return a reference)
s
If I run deployment via api how you can poll the result And also what to use if I make http request I want result back in same response by waiting till execution
@Marvin If I run deployment via api how you can poll the result And also what to use if I make http request I want result back in same response by waiting till execution
m
✅ thought for 122.2 seconds
Yes — if you call a deployment via the Prefect HTTP API, you need to poll separately. The create endpoint does not wait for the flow to finish and does not return the final result in that same response. What the API does 1. 
POST /deployments/{deployment_id}/create_flow_run
2. returns a 
flow_run_id
3. you poll 
GET /flow_runs/{flow_run_id}
4. once state is final, read the result So the pattern is:
Copy code
POST /deployments/{deployment_id}/create_flow_run
-> returns flow_run_id
then:
Copy code
GET /flow_runs/{flow_run_id}
until 
state.type
is one of: - 
COMPLETED
- 
FAILED
- 
CRASHED
- 
CANCELLED
Then inspect the state/result. Important: there is no built-in single HTTP endpoint in Prefect Cloud/Server that blocks until execution completes and then returns the flow’s return value in the same HTTP response. If you want that behavior, you should build a wrapper service around Prefect. Best option if you want one HTTP request that waits and returns the result Create your own API endpoint that calls Prefect’s Python SDK 
run_deployment()
and waits. Example:
Copy code
from fastapi import FastAPI, HTTPException
from prefect.deployments import run_deployment

app = FastAPI()

@app.post("/run")
def run_flow(payload: dict):
    flow_run = run_deployment(
        name="my-flow/my-deployment",
        parameters=payload,
        timeout=None,   # wait until done
        poll_interval=2
    )

    state = flow_run.state

    if state.is_completed():
        return {"flow_run_id": str(flow_run.id), "result": state.result()}

    if state.is_failed() or state.is_crashed() or state.is_cancelled():
        raise HTTPException(status_code=500, detail=str(state))

    return {"flow_run_id": str(flow_run.id), "state": str(state.type)}
That gives your caller a synchronous HTTP experience, while Prefect still runs the deployment remotely. If you must use only raw HTTP against Prefect API Then implement polling yourself. Example polling flow:
Copy code
import time
import requests

PREFECT_API = "<https://api.prefect.cloud/api/accounts/.../workspaces/...>"
API_KEY = "YOUR_API_KEY"
DEPLOYMENT_ID = "YOUR_DEPLOYMENT_ID"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

# Start flow run
resp = <http://requests.post|requests.post>(
    f"{PREFECT_API}/deployments/{DEPLOYMENT_ID}/create_flow_run",
    headers=headers,
    json={"parameters": {"x": 1}},
)
resp.raise_for_status()
flow_run = resp.json()
flow_run_id = flow_run["id"]

# Poll
while True:
    r = requests.get(
        f"{PREFECT_API}/flow_runs/{flow_run_id}",
        headers=headers,
    )
    r.raise_for_status()
    flow_run = r.json()
    state = flow_run["state"]
    state_type = state["type"]

    if state_type in ["COMPLETED", "FAILED", "CRASHED", "CANCELLED"]:
        print(flow_run)
        break

    time.sleep(2)
One caveat on results The flow return value is not always ideal to fetch directly via plain raw HTTP, especially for larger or persisted results. In Python, 
state.result()
is the most reliable way because it knows how to resolve persisted results from storage. So in practice: - Need simple synchronous request/response? Use your own API + 
run_deployment()
- Need pure Prefect HTTP API? Use 
create_flow_run
+ poll 
flow_runs/{id}
- Need large outputs? Better to store the result externally and return a reference/URL/id Useful references: - Prefect deployment triggering/result handling is exposed in the Python SDK via 
run_deployment()
/ 
arun_deployment()
- Prefect API concepts: https://docs.prefect.io/ - Python SDK reference: https://docs.prefect.io/v3/api-ref/python/prefect-deployments-flow_runs If you want, I can give you a complete FastAPI wrapper or Node.js polling example.
s
@Marvin which files consume more space on prefect server If am going to Persist result for few flows and deployment run every 5 min. How to maintain the onpremise server to keep it running
m
✅ thought for 102.0 seconds
On a self-hosted Prefect server, the biggest disk consumers are usually: 1. Database 2. Persisted results 3. Logs / event history If your deployment runs every 5 minutes, the database growth is usually the first thing to watch. What consumes space most 1. Database (
prefect.db
for SQLite, or your Postgres DB if using Postgres) This stores: - flow runs - task runs - state history - logs - events - artifacts metadata For frequent runs, events and logs grow very fast. 2. Persisted results If you enable result persistence, returned flow/task data is stored somewhere. In local/on-prem setups, this is often under Prefect home storage, commonly: - 
~/.prefect/storage/
If the flow returns large payloads every 5 min, this can grow quickly. 3. Logs Verbose logging from tasks and flows can make the DB much larger over time. Default important locations - Prefect home: 
~/.prefect/
- SQLite DB default: 
~/.prefect/prefect.db
- Local result storage default: 
~/.prefect/storage/
So if you are using defaults, the main files/directories to monitor are:
Copy code
~/.prefect/prefect.db
~/.prefect/storage/
For your case: every 5 minutes That means: - 12 runs/hour - 288 runs/day - 8,640 runs/month If each run persists results and writes logs, storage can grow fast. Best practices to keep on-prem healthy 1. Use PostgreSQL for production Do not rely on SQLite for a busy on-prem server. Use: - PostgreSQL for Prefect API database - external object storage for results if possible 2. Do not persist large results locally unless needed If you only need results for a short time, either: - avoid persistence when possible - store only small summaries - write full outputs to S3 / MinIO / shared object storage instead 3. Enable database cleanup / vacuum services Prefect has DB vacuum services for old data cleanup. Recommended settings for a high-frequency environment:
Copy code
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events,flow_runs
PREFECT_EVENTS_RETENTION_PERIOD=2d
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=7d
That means: - events older than 2 days cleaned up - old flow/task run records cleaned after 7 days 4. Keep result retention short If you persist results, set a cleanup policy yourself. Prefect won’t magically manage your local result files forever unless your storage lifecycle handles it. If using local filesystem storage, you may need a cron job to remove old files. 5. Reduce log volume Avoid excessive 
<http://logger.info|logger.info>()
in loops or large object dumps. 6. Monitor disk regularly Check:
Copy code
du -sh ~/.prefect/*
and for DB size:
Copy code
ls -lh ~/.prefect/prefect.db
or for Postgres, monitor DB/table sizes directly. Recommended architecture for your use case If deployments run every 5 min and results matter: - Prefect server DB: PostgreSQL - Result storage: S3 / MinIO / network object storage - Vacuum enabled: yes - Short retention: yes - Local disk: only for temporary server files, not long-term results Practical rule of thumb - Biggest file if using SQLite: 
~/.prefect/prefect.db
- Biggest directory if persisting results locally: 
~/.prefect/storage/
What I’d recommend for on-prem If this is production-like: 1. move from SQLite to Postgres 2. store persisted results outside the server disk 3. enable Prefect DB vacuum 4. set short retention for events/runs 5. add OS-level monitoring for disk + memory + DB growth Useful docs: - Database maintenance - Self-hosted Prefect If you want, I can give you a sample docker-compose setup for on-prem Prefect with Postgres + MinIO + retention settings.
s
@Marvin how prefect worker pulls docker images if deployment has docker image job variable
m
✅ thought for 140.4 seconds
If a deployment has an 
image
in 
job_variables
, the worker uses that image when creating the infrastructure job. But who actually pulls the image depends on the worker type. Docker worker For a Docker work pool / Docker worker: - Prefect worker reads the deployment job variables - sees 
image=...
- uses the Docker API on the worker host - Docker pulls the image if needed - then starts the container from that image So for Docker workers, the pull is effectively done through the worker via Docker, on the machine where the worker is running. If the image is private, you usually provide registry credentials. Typical deployment override:
Copy code
job_variables={
  "image": "my-registry/my-image:1.2.3"
}
For private registries with Docker worker, you may also need registry credentials in the work pool / deployment config. Kubernetes worker For a Kubernetes work pool / Kubernetes worker: - Prefect worker creates a Kubernetes Job manifest - the manifest includes the 
image
- Kubernetes pulls the image on the cluster node So in this case, Prefect does not pull the image itself. Kubernetes does. If the image is private, you need Kubernetes 
imagePullSecrets
. Example conceptually:
Copy code
job_variables={
  "image": "my-registry/my-image:1.2.3",
  "image_pull_secrets": ["my-registry-secret"]
}
Important rule 
job_variables
on the deployment override the work pool defaults. So if your work pool default image is:
Copy code
prefecthq/prefect:3-latest
and the deployment has:
Copy code
job_variables={
  "image": "custom/app:5"
}
then the worker will use 
custom/app:5
for that deployment run. What machine needs network access? Depends on worker type: - Docker worker: the worker host must be able to reach the registry - Kubernetes worker: the Kubernetes nodes must be able to reach the registry What about pull policy? Usually controlled by something like 
image_pull_policy
. Common behavior: - 
Always
→ always pull - 
IfNotPresent
→ pull only if missing - Docker worker may auto-decide based on tag like 
latest
In short - 
job_variables["image"]
tells Prefect what image to run - Docker worker: Docker on the worker host pulls it - Kubernetes worker: Kubernetes pulls it - for private images, configure the proper credentials for that infrastructure - deployment 
job_variables
override work pool defaults Docs: - Work pools and workers - Docker deployment infrastructure - Kubernetes deployment infrastructure If you want, I can show you the exact configuration for: - Docker worker + private registry - Kubernetes worker + imagePullSecrets - or prefect.yaml job_variables example
s
@Marvin To run a docker worker in private network which urls to whitelist to make it running without connectivity issue outbound
m
✅ thought for 94.8 seconds
For a Docker worker in a private network, the minimum outbound allowlist depends on whether you're using Prefect Cloud or self-hosted Prefect Server. If using Prefect Cloud Allow outbound HTTPS to: - 
api.prefect.cloud
- 
app.prefect.cloud
(not needed by the worker itself, but useful for human UI access) - 
<http://auth.workos.com|auth.workos.com>
(mainly for browser/user auth; worker itself primarily needs API access) For the worker itself, the critical one is: - 
api.prefect.cloud:443
That is used for: - polling work from the work pool - creating/updating flow run state - sending logs/events - worker heartbeats If using self-hosted Prefect Server Allow outbound HTTPS to your Prefect API URL, for example: - 
prefect.mycompany.internal:443
or wherever your 
PREFECT_API_URL
points. The worker mainly needs access to: - your configured 
PREFECT_API_URL
Also whitelist image registry endpoints Since this is a Docker worker, it also needs outbound access to wherever the image is pulled from. Examples: - Docker Hub: 
<http://registry-1.docker.io|registry-1.docker.io>
- GitHub Container Registry: 
<http://ghcr.io|ghcr.io>
- AWS ECR: 
<account>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
- Azure ACR: 
<registry>.<http://azurecr.io|azurecr.io>
- GCP Artifact Registry: 
<region>-docker.pkg.dev
If your deployment image is private, also allow the auth flow for that registry as needed. If your code or results use external storage, whitelist those too Common extras: - GitHub source code: 
<http://github.com|github.com>
, 
<http://api.github.com|api.github.com>
- S3: 
<http://s3.amazonaws.com|s3.amazonaws.com>
or regional S3 endpoint - GCS: 
<http://storage.googleapis.com|storage.googleapis.com>
- Azure Blob: 
<account>.<http://blob.core.windows.net|blob.core.windows.net>
- private package index / PyPI: 
<http://pypi.org|pypi.org>
, 
<http://files.pythonhosted.org|files.pythonhosted.org>
So the real allowlist depends on: - where Prefect API is - where Docker images are hosted - where flow code is fetched from - where results/artifacts are stored Practical minimums Prefect Cloud + Docker Hub image
Copy code
api.prefect.cloud:443
<http://registry-1.docker.io:443|registry-1.docker.io:443>
<http://auth.docker.io:443|auth.docker.io:443>
<http://production.cloudflare.docker.com:443|production.cloudflare.docker.com:443>
Self-hosted Prefect + private registry
Copy code
<your-prefect-api-host>:443
<your-registry-host>:443
Recommended checklist Your Docker worker host should be able to reach: 1. Prefect API 2. container registry 3. code storage if using remote source 4. result storage if persisting results remotely 5. package repositories if installing dependencies at runtime Notes - Use DNS/domain allowlisting, not static IPs, especially for Prefect Cloud - Websocket/long-poll behavior is still over standard HTTPS/WSS, usually port 
443
- If you use a corporate proxy, set 
HTTPS_PROXY
/ 
HTTP_PROXY
You can verify the configured API target with:
Copy code
prefect config view
Docs: - Workers - Work pools - Self-hosted Prefect If you want, I can give you a precise firewall allowlist table for your setup if you tell me: - Prefect Cloud or self-hosted - Docker Hub / ECR / ACR / GHCR - Git-based source or baked-in image - S3/GCS/Azure/local results
s
@Marvin prefect server onpremise, code from GitHub, images from dockerhub, deployment stores result in supabase via api,
m
✅ thought for 8.6 seconds
For your setup, the Docker worker needs outbound HTTPS access to these destinations: Required - Your Prefect server API host Example: 
prefect.mycompany.internal:443
This is whatever 
PREFECT_API_URL
points to. - GitHub - 
<http://github.com:443|github.com:443>
- 
<http://api.github.com:443|api.github.com:443>
Needed if your deployment pulls code from GitHub. - Docker Hub - 
<http://registry-1.docker.io:443|registry-1.docker.io:443>
- 
<http://auth.docker.io:443|auth.docker.io:443>
- 
<http://production.cloudflare.docker.com:443|production.cloudflare.docker.com:443>
Needed to pull Docker images. - Supabase - 
<your-project>.<http://supabase.co:443|supabase.co:443>
Needed because your flow writes results to Supabase via API. Possibly required depending on your GitHub auth method If you clone private GitHub repos or use release/archive downloads, also allow: - 
<http://codeload.github.com:443|codeload.github.com:443>
Optional but common - 
<http://pypi.org:443|pypi.org:443>
- 
<http://files.pythonhosted.org:443|files.pythonhosted.org:443>
Only needed if your container installs packages at runtime. Optional Prefect telemetry If you want to disable extra telemetry/noise: - set 
DO_NOT_TRACK=1
- set 
PREFECT_SERVER_ANALYTICS_ENABLED=false
That can reduce calls to telemetry endpoints. Recommended allowlist for your case
Copy code
<your-prefect-api-host>:443
<http://github.com:443|github.com:443>
<http://api.github.com:443|api.github.com:443>
<http://codeload.github.com:443|codeload.github.com:443>
<http://registry-1.docker.io:443|registry-1.docker.io:443>
<http://auth.docker.io:443|auth.docker.io:443>
<http://production.cloudflare.docker.com:443|production.cloudflare.docker.com:443>
<your-project>.<http://supabase.co:443|supabase.co:443>
Important note Because you're self-hosted, the worker does not need 
api.prefect.cloud
unless you are also using Prefect Cloud somewhere else. If GitHub repo is private You also need credentials/token handling inside the worker/container, but network-wise the above hosts are the key ones. Best practice Allow by domain name rather than fixed IPs, especially for: - GitHub - Docker Hub - Supabase Their IPs can change. Quick connectivity checks from the worker host
Copy code
curl -I <https://github.com>
curl -I <https://api.github.com>
curl -I <https://registry-1.docker.io/v2/>
curl -I https://<your-project>.<http://supabase.co|supabase.co>
curl -I https://<your-prefect-api-host>/api/health
If you want, I can turn this into: - a network/firewall checklist - a Docker worker readiness script - or a minimal outbound policy document for your infra team
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by