<@ULVA73B9P> I have prefect 3 running locally on s...
# ask-marvin
c
@Marvin I have prefect 3 running locally on server A (windows server), and I want an agent to run on server B (ubuntu) so I can coordinate tasks between servers A and B. how can i do this?
m
thought for 462.1 seconds
Great question. In Prefect 3.x, “agents” were replaced by “workers.” To run flows on Server B (Ubuntu) while your Prefect Server is on Server A (Windows), you’ll: - Expose the Prefect Server API on A - Point a worker on B at that API - Create a work pool - Deploy flows that target that pool Here’s a concise setup you can follow. 1) Make sure your Prefect Server on A is reachable - Start or restart the server to listen on the network interface and open the firewall for port 4200:
Copy code
prefect server start --host 0.0.0.0 --port 4200
- From B, you should be able to reach A’s API at http//&lt;A IP&gt;4200/api Docs: Prefect Server, Server CLI 2) On B (Ubuntu), point Prefect at A’s API and create a work pool - Install Prefect and set the API URL:
Copy code
pip install -U prefect
export PREFECT_API_URL="http://<A-IP>:4200/api"
- Create a process work pool (you can also do this in the UI):
Copy code
prefect work-pool create --type process ubuntu-pool
- Start a worker on B that polls that pool:
Copy code
prefect worker start --pool ubuntu-pool
Docs: Workers, Settings &amp; Profiles, Work pool CLI, Worker CLI 3) Deploy a flow that targets the Ubuntu pool Important: The worker on B must be able to fetch your flow code. Put your flow in a Git repo or other remote storage accessible from B. Example using Python API (correct for Prefect 3.x):
Copy code
from prefect import flow, Flow

# your flow code lives in a repo accessible to B
# entrypoint is path/to/file.py:flow_function_name in that repo
Flow.from_source(
    source="<https://github.com/your-org/your-repo.git>",
    entrypoint="flows/my_flow.py:my_flow",
).deploy(
    name="run-on-ubuntu",
    work_pool_name="ubuntu-pool",
)
Then trigger the deployment from the UI or CLI:
Copy code
prefect deployments run "my_flow/run-on-ubuntu"
Alternatively, you can use
prefect deploy
with a project and prefect.yaml:
Copy code
prefect deploy ./flows/my_flow.py:my_flow --name run-on-ubuntu
Docs: Deploy with prefect.yaml Coordinating work across A and B - A single flow run executes on one machine (the one running the worker for its pool). - To “coordinate” steps across machines/OSes, create separate deployments targeting different pools and have one flow trigger another via
run_deployment
. Example: a Windows flow triggers a Linux subflow and waits for it
Copy code
from prefect import flow
from prefect.deployments import run_deployment

@flow
def windows_controller():
    # do Windows-specific work here…
    # then trigger a deployment that runs on B's ubuntu-pool
    run = run_deployment(
        name="linux_flow/run-on-ubuntu",  # "<flow_name>/<deployment_name>"
        parameters={"param": "value"},
    )
    # continue after the Linux subflow completes
    return run.state

if __name__ == "__main__":
    windows_controller()
Docs: Deployments Quick checklist - A’s server reachable from B at http//&lt;A IP&gt;4200/api - Firewall allows inbound 4200 to A - On B:
PREFECT_API_URL
set, pool created, worker started - Flow code accessible to B (Git, object storage, etc.) - Deployments point to the correct work_pool_name Would you like help: - Configuring your flow storage (Git/S3/etc.)? - Running a Windows worker on A as well? - Setting this up as a systemd service on B so the worker runs on boot?
c
@Marvin in trying to test connection on B to http//&lt;A IP&gt;4200/api, it gets timed out.
m
thought for 54.1 seconds
Got it — a timeout means B can’t reach A’s Prefect Server over the network. Here’s how to fix and verify connectivity. On Server A (Windows) 1) Start Prefect Server bound to all interfaces - This ensures it listens beyond localhost:
Copy code
prefect server start --host 0.0.0.0 --port 4200
Tip: Watch the startup logs; you should see it listening on 0.0.0.0:4200. 2) Open Windows Firewall for port 4200 (TCP) - Run in an elevated PowerShell or CMD:
Copy code
netsh advfirewall firewall add rule name="Prefect Server 4200" dir=in action=allow protocol=TCP localport=4200
- If there’s any upstream firewall, open 4200 there too. 3) Confirm the server is listening - PowerShell:
Copy code
netstat -ano | findstr :4200
You should see a LISTENING entry on 0.0.0.0:4200. On Server B (Ubuntu) 4) Test raw connectivity first - From B:
Copy code
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
# or
nc -vz <A-IP> 4200
If these time out, it’s still a network/firewall/routing issue. 5) Once reachable, point Prefect at A’s API and test
Copy code
export PREFECT_API_URL="http://<A-IP>:4200/api"
prefect config view | grep PREFECT_API_URL
You can also try loading http//&lt;A IP&gt;4200 in a browser from B’s network to confirm the UI is reachable. Common culprits - Server not bound to 0.0.0.0 (default is often localhost) - Windows Firewall blocking inbound 4200 - A and B are on different networks/VLANs/subnets without routing - Server A running Prefect inside WSL/Docker without proper port exposure If you cannot open the firewall - Use a tunnel (e.g., SSH/Tailscale/ZeroTier) to expose A:4200 to B. - Or run Prefect Server on B and point both machines to it. Docs for reference: - Server: Prefect Server - Server CLI: Server CLI - Settings: Settings &amp; Profiles If it still times out after trying the above, can you share: - The exact command you used to start the server on A - Whether http//&lt;A IP&gt;4200 loads from another machine’s browser - Whether A and B are on the same LAN/subnet (and any known firewalls/security groups)
n
are you trying to share a db between api servers, or do you truly have 2 separate prefect instances?
c
@Nate Hi - I'm a bit of a luddite so I'm not sure 100% what your question means, but I think it's the former - I have deployments I want to create on server B (ubuntu) that I will need to coordinate with other deployments on server A (windows server). My destination database for these operations is on server A (postgres) and prefect.db sits in postgres on server A as well. It doesn't seem like i need 2 separate prefect instances.
I've got the inbound rules on server A allowed for port 4200, and my tests for connectivity on server B keep timing out using curl, telnet, or nc
n
sorry im not sure im clear on your db situation, i might be misreading can you clarify • you intend to use postgres (and not
prefect.db
, which is sqlite) • you intend to share this postgres db between 2 prefect server instances is that right?
c
1. Yes, today, server A's prefect database is in postgres, as I moved it off of sqlite. 2. I think yes, if that allows me to trigger deployments on B from A, and vice versa. Would rather have the logging and such in the same postgres db.
n
well to be clear, if you're sharing db's, im not sure what A and B actually refer to if A and B are just separate instances of
prefect server start
where each use the same
PREFECT_SERVER_DATABASE_URL
, then they will present/have access to the same underlying data running many api servers with the same db is generally just for HA there's no data encapsulation or isolation you get by running 2 servers that talk to the same db
c
Thanks, definitely not HA related. Let's back up then - I have a pipeline that i need to coordinate the deployments of from server A (windows) and server B (ubuntu). They'll need to ping pong back and forth a few times to get the result I need, as in: 1. run server A deployment; 2. after #1 do server B deployment; 3. after #2 do server A deployment, and so on. What I'm stuck on right now is just getting server B to talk to A per the guidance marvin/chatgpt have said. Checking now to make sure there isn't a vanilla networking thing that I'm missing that's blocking the connection.
n
do you actually need communication between servers? or do you just need 2 (client-side) workers, one for each platform? again, i might not understand exactly your requirements but to me this sounds like • one
prefect server start
with a api URL that all workers can have network access to • 2 workers
prefect worker start
prefect worker start --pool ubuntu-pool
(on machine 1) ◦
prefect worker start --pool windows-pool
(on machine 2)
c
Believe it's the latter/your bullets, that I just need the two workers.
n
great! yea workers can run anywhere that they have network access to the server, which sounds useful for your case here
c
prefect config view on server B does indicate now that I'm connected to <server A IP>:4200, so I may be close here...
@Marvin In trying to create the worker-pool from server B (ubuntu), the command hangs infinitely. If I shut down prefect server on A, and do a simple test with a temp web server on A with port 4200, I can connect to that on B. However when starting prefect server again on A, I still can't connect to it on B and continue to time out.
m
thought for 65.5 seconds
Thanks for the detail — that helps narrow it down. Since a simple temp server on A:4200 is reachable from B, but Prefect Server on A is not, this points to Prefect Server binding or environment settings on A (not a general network/firewall issue). Let’s verify and fix binding on Server A (Windows) 1) Check Prefect’s server host/port settings on A - If these are set to localhost, Prefect will only listen on loopback.
Copy code
prefect config view | findstr PREFECT_SERVER_API_
If you see
PREFECT_SERVER_API_HOST=127.0.0.1
(or similar), override it. 2) Explicitly bind the server to all interfaces and a known port - Either set settings persistently:
Copy code
prefect config set PREFECT_SERVER_API_HOST=0.0.0.0 PREFECT_SERVER_API_PORT=4200
Then start:
Copy code
prefect server start
- Or pass them on the command:
Copy code
prefect server start --host 0.0.0.0 --port 4200
Docs: Server CLI 3) Confirm the process is listening on 0.0.0.0:4200 - PowerShell/CMD:
Copy code
netstat -ano | findstr :4200
You should see a LISTENING entry on 0.0.0.0:4200 (note the PID). Optionally:
Copy code
tasklist /FI "PID eq <PID>"
4) Test from A itself - If this fails, binding is still wrong:
Copy code
curl -I <http://127.0.0.1:4200>
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
5) Retest from B (Ubuntu)
Copy code
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
If it’s still not reachable, check these edge cases - WSL or Docker: Are you running
prefect server start
inside WSL/Docker? If yes, the port you tested with the temp web server may have been on native Windows, while Prefect is running inside WSL/Docker and not published externally. In that case you’ll need to publish/forward the port: - Docker: run with
-p 4200:4200
- WSL2: use a Windows portproxy to forward Windows host port to the WSL IP:
Copy code
netsh interface portproxy add v4tov4 listenaddress=<A-IP> listenport=4200 connectaddress=<WSL-IP> connectport=4200
- Profile overrides: Ensure no other profile or environment variables are overriding server host/port. Re-check with
prefect config view
and confirm the settings you expect are active. - IPv6-only bind: Try binding specifically to your IPv4 address:
Copy code
prefect server start --host <A-IP> --port 4200
Once A is reachable, the worker-pool command on B will stop hanging On B:
Copy code
export PREFECT_API_URL="http://<A-IP>:4200/api"
prefect work-pool create --type process ubuntu-pool
prefect worker start --pool ubuntu-pool
Docs: Workers, Work pool CLI If it still hangs, please share: - The exact command and logs you use to start the server on A (including the first few lines that show the bound URL) - Output of
prefect config view | findstr PREFECT_SERVER_API_
on A - Output of
netstat -ano | findstr :4200
on A - Whether Prefect Server is running in WSL or Docker on A We’ll get you connected from there.
c
@Marvin if prefect config view gives me this, what's that tell you: prefect config view 🚀 you are connected to: http://127.0.0.1:4200 PREFECT_PROFILE='local' PREFECT_API_DATABASE_CONNECTION_URL='********' (from profile) PREFECT_API_URL='http://127.0.0.1:4200/api' (from profile) PREFECT_SERVER_EVENTS_RETENTION_PERIOD='P90D' (from profile)
m
thought for 35.3 seconds
That output tells me your Prefect client is pointed at localhost (127.0.0.1) and not at your Windows server’s IP. - The “rocket” line and
PREFECT_API_URL='<http://127.0.0.1:4200/api>'
mean “talk to a Prefect Server on this same machine.” - If you ran this on Server B, it explains the hang: B is trying to reach a Prefect Server on B itself (which isn’t running). - If you ran this on Server A, it also suggests your server may only be bound to loopback unless you start it with a host that’s reachable from B. Fixes On Server B (Ubuntu), point the client at A’s API:
Copy code
prefect config set PREFECT_API_URL=http://<A-IP>:4200/api
prefect config view   # should now show the rocket pointing at http://<A-IP>:4200
On Server A (Windows), ensure the server listens beyond localhost:
Copy code
prefect server start --host 0.0.0.0 --port 4200
- Confirm it’s listening:
Copy code
netstat -ano | findstr :4200
- Open the Windows firewall if needed:
Copy code
netsh advfirewall firewall add rule name="Prefect Server 4200" dir=in action=allow protocol=TCP localport=4200
Verify - From A:
Copy code
curl -I <http://127.0.0.1:4200>
curl -I http://<A-IP>:4200/api
- From B:
Copy code
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
Once
prefect config view
on B shows
http://<A-IP>:4200/api
and you can curl it from B,
prefect work-pool create
and
prefect worker start
should no longer hang. Docs for reference: - Server CLI: Server CLI - Settings & Profiles: Settings &amp; Profiles - Workers: Workers Quick check: Which machine produced that
prefect config view
output? If it was B, updating
PREFECT_API_URL
there should resolve the hang. If it was A, make sure you’re starting the server with
--host 0.0.0.0
so B can reach it.
c
@Marvin Sorry, it was A I ran that on. I am starting with --host 0.0.0.0 and --port 4200 on A already.
m
thought for 53.8 seconds
Thanks — that helps. Since you already start on A with --host 0.0.0.0 and --port 4200 but B still times out (while a temp web server on 4200 is reachable), the most likely causes are: - Prefect Server is not actually listening on 0.0.0.0 (e.g., bound to 127.0.0.1) - Prefect is running inside WSL or Docker on A and the port isn’t exposed to the Windows host network Please run these quick checks on A: 1) What address is Prefect actually bound to? - Start the server and capture the first lines of output that show where it’s serving:
Copy code
prefect server start --host 0.0.0.0 --port 4200
Copy/paste the line that says “Server running at …”. 2) Confirm the listening address at the OS level
Copy code
netstat -ano | findstr :4200
- You want LISTENING on 0.0.0.0:4200 (or on your specific IPv4, e.g., 10.x.x.x:4200). - If it shows 127.0.0.1:4200, the server is only bound to loopback despite the flags. 3) Try binding directly to the IPv4 on A Replace <A-IP> with A’s actual IPv4:
Copy code
prefect server start --host <A-IP> --port 4200
4) Test locally on A
Copy code
curl -I <http://127.0.0.1:4200>
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
All three should respond quickly. 5) If you’re using WSL or Docker on A - WSL2: 0.0.0.0 binds inside the Linux VM only. Forward Windows host port to the WSL IP: - Get WSL IP (inside WSL):
hostname -I
(pick the eth0 IPv4) - On Windows (Admin PowerShell/CMD):
Copy code
netsh interface portproxy add v4tov4 listenaddress=<A-IP> listenport=4200 connectaddress=<WSL-IP> connectport=4200
- Docker: run the server container with
-p 4200:4200
so Windows host exposes it. 6) Re-test from B
Copy code
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
Then:
Copy code
export PREFECT_API_URL="http://<A-IP>:4200/api"
prefect work-pool create --type process ubuntu-pool
If it still fails, please share: - The exact “Server running at …” line from
prefect server start
on A - Output of
netstat -ano | findstr :4200
on A - Whether Prefect is running in native Windows, WSL, or Docker Docs: Server CLI
c
@Marvin 1) Here's what I get: _ _ _ _ _ _ _____ | _ \ _ \ __| __| __/ _| _| | _/ / _|| _|| | (_ | | |_| |_|_\___|_| |___\___| |_| Configure Prefect to communicate with the server with: prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api View the API reference documentation at http://0.0.0.0:4200/docs Check out the dashboard at http://0.0.0.0:4200 2) confirmed it's listening 3) I'll try binding directly to <IP A>
m
thought for 54.1 seconds
Thanks — that banner confirms Prefect thinks it’s bound to 0.0.0.0:4200. Since you can reach a temp server on 4200 from B but not Prefect, the next most likely issue is environment isolation (e.g., Prefect running inside WSL/Docker on A) or the OS binding not matching the banner. Do these quick checks: 1) Bind to the specific IPv4 on A
Copy code
prefect server start --host <A-IP> --port 4200
Then on A:
Copy code
curl -I <http://127.0.0.1:4200>
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
And from B:
Copy code
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
2) Confirm the actual listener on A
Copy code
netstat -ano | findstr :4200
Look for LISTENING on <A-IP>:4200 or 0.0.0.0:4200. If it shows 127.0.0.1:4200, it’s loopback-only despite flags. 3) Sanity check with a “same environment” test Run a temp web server in the EXACT environment you run Prefect: - If Prefect is in a Windows venv, use that Python. - If Prefect is in WSL, run this inside WSL. - If Prefect is in Docker, run inside the container.
Copy code
python -c "import http.server, socketserver; socketserver.TCPServer(('0.0.0.0', 4200), http.server.SimpleHTTPRequestHandler).serve_forever()"
Then try from B:
Copy code
curl -I http://<A-IP>:4200
If this times out, the issue is environment-level isolation (WSL/Docker networking), not Prefect itself. If Prefect is in WSL or Docker on A - WSL2 port forward from Windows host to the WSL IP:
Copy code
# Inside WSL: get IP
hostname -I
# On Windows (Admin), forward port 4200 from Windows to WSL
netsh interface portproxy add v4tov4 listenaddress=<A-IP> listenport=4200 connectaddress=<WSL-IP> connectport=4200
- Docker: run the server container with published port
Copy code
docker run -p 4200:4200 <your-prefect-server-image>
Firewall scope on Windows - Ensure the inbound rule applies to the active profile (Domain/Private/Public) and isn’t restricted to a different interface. - You can create a program-based allow rule for your Python executable or an inbound port rule for TCP 4200 across all profiles. Once B can curl http//&lt;A IP&gt;4200/api, set B’s client and proceed:
Copy code
prefect config set PREFECT_API_URL=http://<A-IP>:4200/api
prefect work-pool create --type process ubuntu-pool
prefect worker start --pool ubuntu-pool
If it still fails, could you share: - Output of netstat on A for :4200 - Whether Prefect is running in native Windows, WSL, or Docker - The result of the “same environment” temp server test Server CLI docs: Server CLI
c
@Marvin prefect was installed in a conda venv
m
thought for 55.8 seconds
Got it — thanks. Since Prefect is running in a conda venv on Windows, the most common cause of “temp server is reachable but Prefect isn’t” is Windows Defender Firewall blocking that specific Python executable (conda’s python.exe), even if the port itself looks open. Do this on Server A (Windows) 1) Confirm which Python and Prefect you’re using - In the same conda env where you run Prefect:
Copy code
where python
where prefect
Note the full path to python.exe (e.g., C:\Users\you\miniconda3\envs\prefect\python.exe). 2) Create a program-based inbound allow rule for that python.exe - Run in an elevated PowerShell or CMD:
Copy code
netsh advfirewall firewall add rule name="Prefect Server (Conda Python)" dir=in action=allow program="C:\path\to\conda\envs\yourenv\python.exe" enable=yes profile=any
- If you previously created an “Allow app through firewall” entry for a different Python, it won’t apply to this conda python.exe. Make sure the rule targets the exact path from step 1. 3) Also ensure a port-based rule exists (belt and suspenders)
Copy code
netsh advfirewall firewall add rule name="Prefect Server 4200" dir=in action=allow protocol=TCP localport=4200
4) Bind explicitly to your IPv4 - Sometimes this clears up any IPv6/host ambiguity:
Copy code
prefect server start --host <A-IP> --port 4200
- Verify the OS-level listener:
Copy code
netstat -ano | findstr :4200
You want LISTENING on <A-IP>:4200 or 0.0.0.0:4200 (not just 127.0.0.1:4200). 5) Sanity-check using the same conda Python - Start a temporary server with that same python.exe:
Copy code
python -c "import http.server, socketserver; socketserver.TCPServer(('0.0.0.0', 4200), http.server.SimpleHTTPRequestHandler).serve_forever()"
- From B:
Copy code
curl -I http://<A-IP>:4200
If this is reachable with the conda python, the firewall rule is good and Prefect should be reachable too. 6) Retest from B and set the client - From B:
Copy code
curl -I http://<A-IP>:4200
curl -I http://<A-IP>:4200/api
- Then:
Copy code
prefect config set PREFECT_API_URL=http://<A-IP>:4200/api
prefect work-pool create --type process ubuntu-pool
prefect worker start --pool ubuntu-pool
If it still times out: - Please share: - Output of
netstat -ano | findstr :4200
on A - The python.exe path from
where python
- Whether the conda-python temp server in step 5 was reachable from B - Also confirm you’re not running Prefect inside WSL or Docker; if you are, you’ll need to port-forward (WSL portproxy or Docker
-p 4200:4200
). References: - Server CLI: Server CLI - Settings & Profiles: Settings &amp; Profiles - Workers: Workers
c
@Marvin yep, it was python.exe blocks set for TCP for whatever reason. can nc into prefect at 4200 now on B.
m
thought for 58.4 seconds
Awesome — glad you found it. Now that B can reach A’s Prefect Server, here’s a quick set of next steps to finish the setup and test end-to-end: 1) On B: point Prefect at A’s API
Copy code
prefect config set PREFECT_API_URL=http://<A-IP>:4200/api
prefect config view   # rocket should point to http://<A-IP>:4200
2) Create a work pool and start a worker on B
Copy code
prefect work-pool create --type process ubuntu-pool
prefect worker start --pool ubuntu-pool
- Leave the worker running; it will poll for flow runs. 3) Deploy a flow targeting that pool Option A: Deploy from source (Python API)
Copy code
from prefect import flow, Flow

@flow
def hello():
    print("Hello from Server B!")

Flow.from_source(
    source="<https://github.com/your-org/your-repo.git>",
    entrypoint="flows/hello.py:hello",
).deploy(
    name="hello-on-b",
    work_pool_name="ubuntu-pool",
)
Option B: Deploy from a local project on A or B - From the project root with a prefect.yaml:
Copy code
prefect deploy ./flows/hello.py:hello --name hello-on-b --pool ubuntu-pool
4) Trigger a run and watch logs
Copy code
prefect deployment run "hello/hello-on-b" --watch
- You should see the run picked up by the worker on B and logs streaming in the UI at http//&lt;A IP&gt;4200. Tips - If you don’t have shared code storage yet, put your flow in a Git repo so B’s worker can fetch it. - For concurrency limits on B: start the worker with
--limit N
. - If you want to start the worker on boot, I can share a systemd unit for Ubuntu. Want me to help write a minimal prefect.yaml or a systemd service for the worker?